64-bit file i/o

[rutiger]

Does anybody know of any freely available code written to handle 64-bit
file i/o for octave? I cant get octave to build using --enable-64
since, as the configure script tells me, on my machine the "pointers
are not 64-bits wide so disabling 64-bit features".

Thanks. Any help at all is greatly appreciated.

octave:56> ver
----------------------------------------------------------------------
GNU Octave Version 2.9.9
GNU Octave License: GNU General Public License, Version 2
Operating System: Linux 2.6.9-42.0.3.ELsmp #1 SMP Mon Sep 25 17:28:02
EDT 2006 i686
----------------------------------------------------------------------

[david bateman]

Post by rutiger
Does anybody know of any freely available code written to handle 64-bit
file i/o for octave? I cant get octave to build using --enable-64
since, as the configure script tells me, on my machine the "pointers
are not 64-bits wide so disabling 64-bit features".
Thanks. Any help at all is greatly appreciated.
octave:56> ver
----------------------------------------------------------------------
GNU Octave Version 2.9.9
GNU Octave License: GNU General Public License, Version 2
Operating System: Linux 2.6.9-42.0.3.ELsmp #1 SMP Mon Sep 25 17:28:02
EDT 2006 i686
----------------------------------------------------------------------

For file I/O you don't need the "--enable-64" option. What --enable-64
does is use 64 bit ints for the matrix indexing. With 32 bit signed
integers for indexing you are limited to 2^31 indexes. Some work has
been done to try and relax constrains so that that is 2^31 per
dimension, but there are sure to be cases where that isn't true, so it
might be a total matrix size of 2^31. 64-bit indexing avoids this and
is available for use on the newer 64 bit machines.

However, you are asking for isn't this, but rather reading 64-bit
values from a file, probably into a uint64 or int64 type.. Try
something like

fid = fopen(file,"rb");
[val, count] = fread(fid,Inf,"uint64");
fclose(fid)

and then see what is in val, with "whos val". Also check the help page
for fread as it allows you to manipulate the fashion in which the data
is read for `different architectures means of storing the data (ie big
and little endian)...

D.

[rutiger]

Post by david bateman

Post by rutiger
Does anybody know of any freely available code written to handle 64-bit
file i/o for octave? I cant get octave to build using --enable-64
since, as the configure script tells me, on my machine the "pointers
are not 64-bits wide so disabling 64-bit features".
Thanks. Any help at all is greatly appreciated.
octave:56> ver
----------------------------------------------------------------------
GNU Octave Version 2.9.9
GNU Octave License: GNU General Public License, Version 2
Operating System: Linux 2.6.9-42.0.3.ELsmp #1 SMP Mon Sep 25 17:28:02
EDT 2006 i686
----------------------------------------------------------------------

For file I/O you don't need the "--enable-64" option. What --enable-64
does is use 64 bit ints for the matrix indexing. With 32 bit signed
integers for indexing you are limited to 2^31 indexes. Some work has
been done to try and relax constrains so that that is 2^31 per
dimension, but there are sure to be cases where that isn't true, so it
might be a total matrix size of 2^31. 64-bit indexing avoids this and
is available for use on the newer 64 bit machines.
However, you are asking for isn't this, but rather reading 64-bit
values from a file, probably into a uint64 or int64 type.. Try
something like
fid = fopen(file,"rb");
[val, count] = fread(fid,Inf,"uint64");
fclose(fid)
and then see what is in val, with "whos val". Also check the help page
for fread as it allows you to manipulate the fashion in which the data
is read for `different architectures means of storing the data (ie big
and little endian)...
D.

I should have been more specific in my question. Sorry about that. The
problem is I/O with extremely large files (tens of gigs in some cases)
and in this case Octave can't even open the file properly. I know that
a work around is possible as we had a fellow at work gin up an fstream
class using 'long long' byte indexing to handle these files when
coding in c++ (I think GNU has since addressed this problem). I'd port
this code over to a binary Octave file but 1) it is proprietary and 2)
I thought someone might have already done something similar.

[rutiger]

Post by rutiger

Post by david bateman

Post by rutiger
Does anybody know of any freely available code written to handle 64-bit
file i/o for octave? I cant get octave to build using --enable-64
since, as the configure script tells me, on my machine the "pointers
are not 64-bits wide so disabling 64-bit features".
Thanks. Any help at all is greatly appreciated.
octave:56> ver
----------------------------------------------------------------------
GNU Octave Version 2.9.9
GNU Octave License: GNU General Public License, Version 2
Operating System: Linux 2.6.9-42.0.3.ELsmp #1 SMP Mon Sep 25 17:28:02
EDT 2006 i686
----------------------------------------------------------------------

For file I/O you don't need the "--enable-64" option. What --enable-64
does is use 64 bit ints for the matrix indexing. With 32 bit signed
integers for indexing you are limited to 2^31 indexes. Some work has
been done to try and relax constrains so that that is 2^31 per
dimension, but there are sure to be cases where that isn't true, so it
might be a total matrix size of 2^31. 64-bit indexing avoids this and
is available for use on the newer 64 bit machines.
However, you are asking for isn't this, but rather reading 64-bit
values from a file, probably into a uint64 or int64 type.. Try
something like
fid = fopen(file,"rb");
[val, count] = fread(fid,Inf,"uint64");
fclose(fid)
and then see what is in val, with "whos val". Also check the help page
for fread as it allows you to manipulate the fashion in which the data
is read for `different architectures means of storing the data (ie big
and little endian)...
D.

I should have been more specific in my question. Sorry about that. The
problem is I/O with extremely large files (tens of gigs in some cases)
and in this case Octave can't even open the file properly. I know that
a work around is possible as we had a fellow at work gin up an fstream
class using 'long long' byte indexing to handle these files when
coding in c++ (I think GNU has since addressed this problem). I'd port
this code over to a binary Octave file but 1) it is proprietary and 2)
I thought someone might have already done something similar.

bump.