| Exhaustive memory allocation using arrays [message #181744] |
Wed, 29 May 2013 06:27  |
tombert.at
Messages: 3
Registered: May 2013
Karma: 0
|
Junior Member |
|
|
Why is the following code using > 100MB?
for ($i=0; $i < 1000000; $i++) {
$value[$i] = 23.4567890123;
}
echo (memory_get_usage(TRUE) / (1024 * 1024)) . "MB <br>";
It looks like that each item takes 100 Byte for allocation??
thx for your help!
|
|
|
|
|
|
|
|
| Re: Exhaustive memory allocation using arrays [message #181747 is a reply to message #181744] |
Wed, 29 May 2013 08:22  |
Denis McMahon
Messages: 634
Registered: September 2010
Karma: 0
|
Senior Member |
|
|
On Tue, 28 May 2013 23:27:13 -0700, tombert.at wrote:
> Why is the following code using > 100MB?
>
> for ($i=0; $i < 1000000; $i++) {
> $value[$i] = 23.4567890123;
> }
> echo (memory_get_usage(TRUE) / (1024 * 1024)) . "MB <br>";
>
> It looks like that each item takes 100 Byte for allocation??
From: http://php.net/manual/en/language.types.array.php
"An array in PHP is actually an ordered map."
You may be expecting arrays where an array of 1e6 floats is held in
memory in 1e6 sequential 8 byte locations, for a total of 8e6 bytes, but
I suspect that if you did, you'd be very very very wrong.
It's possible that:
$arr = array_values( $arr );
May shrink your array, I don't know for sure.
It's also possible that:
$a = Array();
for ( $i = 0; i < 1e6; i++ )
$a[] = 23.4567890123;
might produce a smaller array, simply because it's not assigning keys.
Again, I don't know for sure.
The reason I don't know for sure is that these implementation details
are, to me, just that. If they don't affect me, I don't need to
understand them in the level of detail you're looking for. There's plenty
of things I do need to understand at that level of detail, and I have
limited time to put into gaining such understandings, so I use it where
it's most needed. :)
--
Denis McMahon, denismfmcmahon(at)gmail(dot)com
|
|
|
|
| Re: Exhaustive memory allocation using arrays [message #181748 is a reply to message #181747] |
Wed, 29 May 2013 08:54 |
The Natural Philosoph
Messages: 993
Registered: September 2010
Karma: 0
|
Senior Member |
|
|
On 29/05/13 09:22, Denis McMahon wrote:
> On Tue, 28 May 2013 23:27:13 -0700, tombert.at wrote:
>
>> Why is the following code using > 100MB?
>>
>> for ($i=0; $i < 1000000; $i++) {
>> $value[$i] = 23.4567890123;
>> }
>> echo (memory_get_usage(TRUE) / (1024 * 1024)) . "MB <br>";
>>
>> It looks like that each item takes 100 Byte for allocation??
> From: http://php.net/manual/en/language.types.array.php
>
> "An array in PHP is actually an ordered map."
>
> You may be expecting arrays where an array of 1e6 floats is held in
> memory in 1e6 sequential 8 byte locations, for a total of 8e6 bytes, but
> I suspect that if you did, you'd be very very very wrong.
>
> It's possible that:
>
> $arr = array_values( $arr );
>
> May shrink your array, I don't know for sure.
>
> It's also possible that:
>
> $a = Array();
> for ( $i = 0; i < 1e6; i++ )
> $a[] = 23.4567890123;
>
> might produce a smaller array, simply because it's not assigning keys.
> Again, I don't know for sure.
>
> The reason I don't know for sure is that these implementation details
> are, to me, just that. If they don't affect me, I don't need to
> understand them in the level of detail you're looking for. There's plenty
> of things I do need to understand at that level of detail, and I have
> limited time to put into gaining such understandings, so I use it where
> it's most needed. :)
>
Exactly so. There are times when you realise that an 'excellent for some
tasks' tool is in fact a complete dogs breakfast when applied to another
task.
I found, for example, that on the fly generation of graphics using PHP
was massively CPU and RAM hungry and appeared to suffer re-rentrancy
problems when used in multiple concurrent access: the solution was to
generate the image data asynchronously using a C program.
I can think of many ways in which an array element dynamically assigned
needing pointers to memory blocks and so on could easily have several
tens of bytes overhead.
That of course is an issue with an interpreted language: it has to more
or less avoid statically allocated memory especially if there is no way
to declare it.
--
Ineptocracy
(in-ep-toc’-ra-cy) – a system of government where the least capable to lead are elected by the least capable of producing, and where the members of society least likely to sustain themselves or succeed, are rewarded with goods and services paid for by the confiscated wealth of a diminishing number of producers.
|
|
|
|
| Re: Exhaustive memory allocation using arrays [message #181750 is a reply to message #181746] |
Wed, 29 May 2013 11:00 |
Christoph Becker
Messages: 91
Registered: June 2012
Karma: 0
|
Member |
|
|
tombert.at wrote:
> And, doubling the memory size by assigning a new value:
>
> for ($i=0; $i < 1000000; $i++) {
> $value[$i] = "23.4567890123";
> }
> for ($i=0; $i < 1000000; $i++) {
> $value[$i] = sprintf("%f", $value[$i]);
> }
>
> Now php is using ~400 MB
I *assume* that the strings in the first loop do not actually allocate a
struct str for every instance, but instead reference a single statically
allocated instance. The second loop is probably allocating a new struct
str for every sprintf().
Compare with:
for ($i=0; $i < 1000000; $i++) {
$value[$i] = "23.4567890123";
}
for ($i=0; $i < 1000000; $i++) {
$value[$i] = floatval($value[$i]);
}
--
Christoph M. Becker
|
|
|
|
| Re: Exhaustive memory allocation using arrays [message #181754 is a reply to message #181744] |
Wed, 29 May 2013 13:30 |
Thomas 'PointedEars'
Messages: 701
Registered: October 2010
Karma: 0
|
Senior Member |
|
|
tombert(dot)at(at)gmail(dot)com wrote:
^^^^^^^^^^^^^^^^^^^^
Please get a real name.
> Why is the following code using > 100MB?
>
> for ($i=0; $i < 1000000; $i++) {
> $value[$i] = 23.4567890123;
> }
> echo (memory_get_usage(TRUE) / (1024 * 1024)) . "MB <br>";
>
> It looks like that each item takes 100 Byte for allocation??
First of all, 1 MB (megabyte) is 1'000'000 (one million, 10⁶) bytes. What
you call “MB” here actually is 1 _MiB_ (mega-binary byte; short: mebibyte)
or 2²⁰ bytes = 1'048'576 bytes (as you divided by 2¹⁰ × 2¹⁰ = 2²⁰ to arrive
at your “MB” value):
<http://en.wikipedia.org/wiki/Mebibyte>
So your calculation
100 “MB” ∕ 1'000'000 items = 100 bytes∕item
is wrong to begin with. Assuming that you got something close to 100 as a
result of
memory_get_usage(TRUE) / (1024 * 1024)
each item would take about
104'857'600 bytes / 1'000'000 = 105 bytes
<http://php.net/memory_get_usage>
Second, the value you assigned is a value of the float type. In PHP, the
precision of the float type, thus the number of bits required to store such
a value, is platform-dependent:
<http://php.net/manual/en/language.types.float.php>
The manual says that IEEE-754 “double precision” floating-point values are a
common implementation (as they are in other equally dynamic programming
languages), but we cannot assume that to be the case on your platform. And
you have not said which platform and PHP version you are using, so we cannot
look into the pertinent source code.
Third, you are not just storing the float value. You are storing a key-
value relationship:
,-<http://www.php.net/manual/en/language.types.array.php>
|
| An array in PHP is actually an ordered map. A map is a type that
| associates values to keys. […]
Fourth, that PHP arrays can be associative, i. e. can have string keys, and
that using the string representation of a numeric key accesses the same item
as the number (ibid.), suggests implementations in which all keys are
actually string values (as it is in other equally dynamic programming
languages, like ECMAScript implementations; if not like that, you would have
to store the type of the key which is even more memory-expensive), and
,-<http://www.php.net/manual/en/language.types.string.php>
|
| The string in PHP is implemented as an array of bytes and an integer
| indicating the length of the buffer. […]
Fast access to array items further suggests that the keys are not only
stored as-is (for array_keys()), but that the implementation includes a hash
table.
Fifth, PHP arrays do have an internal pointer – <http://php.net/current> –
that can be advanced: <http://php.net/next>. Combined with the dynamic size
of PHP arrays this suggests that the items of an array are at least arranged
in a linked list. The existence of a function to rewind the internal
pointer – <http://php.net/prev> – further suggests that they are arranged in
a double-linked list.
Finally, you are not just measuring the memory that is needed for the array,
but for executing the preceding program. There is *at least* the memory for
$i variable to consider. So “your math does not add up”.
Probably there are even more factors to consider. See also the
memory_get_usage() documentation and user comments.
So consider this: A PHP array could be implemented *at least* as a hash
table (for quick item access) in which the string keys are stored in a
linked list (so that array_keys() works), and the stored values could be in
a double-linked list (so that internal pointer movements work). In this
hash table you are storing items whose plain values require *at least* 64
bits = 8 bytes of memory each. Add to that the memory required to run the
program that fills the array.
I know of no native way to determine the memory it takes to store the PHP
array; it can perhaps be obtained with a PHP profiler. In any case, I
presume it will be smaller than what you measured with memory_get_usage().
HTH
PointedEars
--
var bugRiddenCrashPronePieceOfJunk = (
navigator.userAgent.indexOf('MSIE 5') != -1
&& navigator.userAgent.indexOf('Mac') != -1
) // Plone, register_function.js:16
|
|
|
|
Calendar
Search
Help
Members
Register
Login
Home
]