c++ Sphinx rt_attr_string truncates data
I'm inserting the content of my documents in an rt_attr_string field. I have declared the max_packet_size as 128M.
However, my input string of 5075850 characters is truncated to a string of 881546 characters. How is this possible and how can I overcome this unexpected behavior? I have already checked the inserted strings. They contain valid characters between a-Z, 0-9 and _ ; or ?
Therefore, why is Sphinx behaving like this? The queried value is directly queried after the insert statement has executed. Therefore, it seems that Sphinx has some issues?
I have logged the query and the queried values, which can be seen in this file
Please note, I insert encrypted values since this is a requirement in my project. My software is perfectly capable of keeping all Sphinx functionalities, but this seems to be a Sphinx related issue since the inserted value differs from the returned value by Sphinx?
The configuration of my index is:
index Test
{
type = rt
path = /mnt/data001/Test
rt_field = Name
rt_field = Extension
rt_field = Content
rt_field = Tags
rt_attr_uint = Reference
rt_attr_uint = FileSize
rt_attr_uint = LastModified
rt_attr_uint = LastModifiedYear
rt_attr_uint = LastModifiedMonth
rt_attr_uint = LastModifiedDay
rt_attr_string = Content
rt_mem_limit = 1024M
charset_table = A..Z, a..z, 0..9, _, ;, %
ondisk_attrs = pool
}
searchd
{
listen = 9306:mysql41
log = /var/log/sphinxsearch/searchd.log
read_timeout = 5
max_children = 30
pid_file = /var/run/searchd.pid
max_packet_size = 128M
binlog_path = /mnt/data001
}
sphinx
asked Nov 15 '18 at 22:18
Thomas van HesterenThomas van Hesteren
5318
|
show 11 more comments
I'm inserting the content of my documents in an rt_attr_string field. I have declared the max_packet_size as 128M.
However, my input string of 5075850 characters is truncated to a string of 881546 characters. How is this possible and how can I overcome this unexpected behavior? I have already checked the inserted strings. They contain valid characters between a-Z, 0-9 and _ ; or ?
Therefore, why is Sphinx behaving like this? The queried value is directly queried after the insert statement has executed. Therefore, it seems that Sphinx has some issues?
I have logged the query and the queried values, which can be seen in this file
Please note, I insert encrypted values since this is a requirement in my project. My software is perfectly capable of keeping all Sphinx functionalities, but this seems to be a Sphinx related issue since the inserted value differs from the returned value by Sphinx?
The configuration of my index is:
index Test
{
type = rt
path = /mnt/data001/Test
rt_field = Name
rt_field = Extension
rt_field = Content
rt_field = Tags
rt_attr_uint = Reference
rt_attr_uint = FileSize
rt_attr_uint = LastModified
rt_attr_uint = LastModifiedYear
rt_attr_uint = LastModifiedMonth
rt_attr_uint = LastModifiedDay
rt_attr_string = Content
rt_mem_limit = 1024M
charset_table = A..Z, a..z, 0..9, _, ;, %
ondisk_attrs = pool
}
searchd
{
listen = 9306:mysql41
log = /var/log/sphinxsearch/searchd.log
read_timeout = 5
max_children = 30
pid_file = /var/run/searchd.pid
max_packet_size = 128M
binlog_path = /mnt/data001
}
sphinx
asked Nov 15 '18 at 22:18
Thomas van HesterenThomas van Hesteren
5318
I can't reproduce your issue. You should create a simple example: an empty Sphinx index and a script which writes a big string into it and then reads it and compares the results.
– hank
Nov 16 '18 at 9:04
@hank, alright. Could you maybe tell me or there are any limits on thert_attr_string? Because, I can't discover a relation with the length of the returned string... (0.8MB at this point)
– Thomas van Hesteren
Nov 16 '18 at 9:13
I don't know exact limits, but in my test I successfully wrote and read back5075850characters.
– hank
Nov 16 '18 at 13:46
@hank, how is that possible? My insert ‘fails’ each try. Do you use my configuration as well?
– Thomas van Hesteren
Nov 16 '18 at 15:40
@hank, could you share your configuration?
– Thomas van Hesteren
Nov 17 '18 at 7:58
|
show 11 more comments
I'm inserting the content of my documents in an rt_attr_string field. I have declared the max_packet_size as 128M.
However, my input string of 5075850 characters is truncated to a string of 881546 characters. How is this possible and how can I overcome this unexpected behavior? I have already checked the inserted strings. They contain valid characters between a-Z, 0-9 and _ ; or ?
Therefore, why is Sphinx behaving like this? The queried value is directly queried after the insert statement has executed. Therefore, it seems that Sphinx has some issues?
I have logged the query and the queried values, which can be seen in this file
Please note, I insert encrypted values since this is a requirement in my project. My software is perfectly capable of keeping all Sphinx functionalities, but this seems to be a Sphinx related issue since the inserted value differs from the returned value by Sphinx?
The configuration of my index is:
index Test
{
type = rt
path = /mnt/data001/Test
rt_field = Name
rt_field = Extension
rt_field = Content
rt_field = Tags
rt_attr_uint = Reference
rt_attr_uint = FileSize
rt_attr_uint = LastModified
rt_attr_uint = LastModifiedYear
rt_attr_uint = LastModifiedMonth
rt_attr_uint = LastModifiedDay
rt_attr_string = Content
rt_mem_limit = 1024M
charset_table = A..Z, a..z, 0..9, _, ;, %
ondisk_attrs = pool
}
searchd
{
listen = 9306:mysql41
log = /var/log/sphinxsearch/searchd.log
read_timeout = 5
max_children = 30
pid_file = /var/run/searchd.pid
max_packet_size = 128M
binlog_path = /mnt/data001
}
sphinx
asked Nov 15 '18 at 22:18
Thomas van HesterenThomas van Hesteren
5318
I'm inserting the content of my documents in an rt_attr_string field. I have declared the max_packet_size as 128M.
However, my input string of 5075850 characters is truncated to a string of 881546 characters. How is this possible and how can I overcome this unexpected behavior? I have already checked the inserted strings. They contain valid characters between a-Z, 0-9 and _ ; or ?
Therefore, why is Sphinx behaving like this? The queried value is directly queried after the insert statement has executed. Therefore, it seems that Sphinx has some issues?
I have logged the query and the queried values, which can be seen in this file
Please note, I insert encrypted values since this is a requirement in my project. My software is perfectly capable of keeping all Sphinx functionalities, but this seems to be a Sphinx related issue since the inserted value differs from the returned value by Sphinx?
The configuration of my index is:
index Test
{
type = rt
path = /mnt/data001/Test
rt_field = Name
rt_field = Extension
rt_field = Content
rt_field = Tags
rt_attr_uint = Reference
rt_attr_uint = FileSize
rt_attr_uint = LastModified
rt_attr_uint = LastModifiedYear
rt_attr_uint = LastModifiedMonth
rt_attr_uint = LastModifiedDay
rt_attr_string = Content
rt_mem_limit = 1024M
charset_table = A..Z, a..z, 0..9, _, ;, %
ondisk_attrs = pool
}
searchd
{
listen = 9306:mysql41
log = /var/log/sphinxsearch/searchd.log
read_timeout = 5
max_children = 30
pid_file = /var/run/searchd.pid
max_packet_size = 128M
binlog_path = /mnt/data001
}
sphinx
sphinx
asked Nov 15 '18 at 22:18
Thomas van HesterenThomas van Hesteren
5318
asked Nov 15 '18 at 22:18
Thomas van HesterenThomas van Hesteren
5318
edited Nov 17 '18 at 12:45
Thomas van Hesteren
asked Nov 15 '18 at 22:18
Thomas van HesterenThomas van Hesteren
5318
asked Nov 15 '18 at 22:18
Thomas van HesterenThomas van Hesteren
5318
asked Nov 15 '18 at 22:18
Thomas van HesterenThomas van Hesteren
5318
5318
I can't reproduce your issue. You should create a simple example: an empty Sphinx index and a script which writes a big string into it and then reads it and compares the results.
– hank
Nov 16 '18 at 9:04
@hank, alright. Could you maybe tell me or there are any limits on thert_attr_string? Because, I can't discover a relation with the length of the returned string... (0.8MB at this point)
– Thomas van Hesteren
Nov 16 '18 at 9:13
I don't know exact limits, but in my test I successfully wrote and read back5075850characters.
– hank
Nov 16 '18 at 13:46
@hank, how is that possible? My insert ‘fails’ each try. Do you use my configuration as well?
– Thomas van Hesteren
Nov 16 '18 at 15:40
@hank, could you share your configuration?
– Thomas van Hesteren
Nov 17 '18 at 7:58
|
show 11 more comments
I can't reproduce your issue. You should create a simple example: an empty Sphinx index and a script which writes a big string into it and then reads it and compares the results.
– hank
Nov 16 '18 at 9:04
@hank, alright. Could you maybe tell me or there are any limits on thert_attr_string? Because, I can't discover a relation with the length of the returned string... (0.8MB at this point)
– Thomas van Hesteren
Nov 16 '18 at 9:13
I don't know exact limits, but in my test I successfully wrote and read back5075850characters.
– hank
Nov 16 '18 at 13:46
@hank, how is that possible? My insert ‘fails’ each try. Do you use my configuration as well?
– Thomas van Hesteren
Nov 16 '18 at 15:40
@hank, could you share your configuration?
– Thomas van Hesteren
Nov 17 '18 at 7:58
I can't reproduce your issue. You should create a simple example: an empty Sphinx index and a script which writes a big string into it and then reads it and compares the results.
– hank
Nov 16 '18 at 9:04
I can't reproduce your issue. You should create a simple example: an empty Sphinx index and a script which writes a big string into it and then reads it and compares the results.
– hank
Nov 16 '18 at 9:04
@hank, alright. Could you maybe tell me or there are any limits on the
rt_attr_string? Because, I can't discover a relation with the length of the returned string... (0.8MB at this point)– Thomas van Hesteren
Nov 16 '18 at 9:13
@hank, alright. Could you maybe tell me or there are any limits on the
rt_attr_string? Because, I can't discover a relation with the length of the returned string... (0.8MB at this point)– Thomas van Hesteren
Nov 16 '18 at 9:13
I don't know exact limits, but in my test I successfully wrote and read back
5075850 characters.– hank
Nov 16 '18 at 13:46
I don't know exact limits, but in my test I successfully wrote and read back
5075850 characters.– hank
Nov 16 '18 at 13:46
@hank, how is that possible? My insert ‘fails’ each try. Do you use my configuration as well?
– Thomas van Hesteren
Nov 16 '18 at 15:40
@hank, how is that possible? My insert ‘fails’ each try. Do you use my configuration as well?
– Thomas van Hesteren
Nov 16 '18 at 15:40
@hank, could you share your configuration?
– Thomas van Hesteren
Nov 17 '18 at 7:58
@hank, could you share your configuration?
– Thomas van Hesteren
Nov 17 '18 at 7:58
|
show 11 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53328679%2fc-sphinx-rt-attr-string-truncates-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53328679%2fc-sphinx-rt-attr-string-truncates-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I can't reproduce your issue. You should create a simple example: an empty Sphinx index and a script which writes a big string into it and then reads it and compares the results.
– hank
Nov 16 '18 at 9:04
@hank, alright. Could you maybe tell me or there are any limits on the
rt_attr_string? Because, I can't discover a relation with the length of the returned string... (0.8MB at this point)– Thomas van Hesteren
Nov 16 '18 at 9:13
I don't know exact limits, but in my test I successfully wrote and read back
5075850characters.– hank
Nov 16 '18 at 13:46
@hank, how is that possible? My insert ‘fails’ each try. Do you use my configuration as well?
– Thomas van Hesteren
Nov 16 '18 at 15:40
@hank, could you share your configuration?
– Thomas van Hesteren
Nov 17 '18 at 7:58