Repository navigation
Conversation
|
The LINUX_X64_RELEASE_NTS failure is an SNMP timeout in ext/snmp/tests/snmp-object-errno-errstr.phpt. It is not caused by this change. |
|
I would propose the following changes in addtion --- a/ext/phar/phar_object.c
+++ b/ext/phar/phar_object.c
@@ -1640,6 +1640,28 @@ static int phar_build(zend_object_iterator *iter, void *puser) /* {{{ */
return ZEND_HASH_APPLY_KEEP;
}
+ if (php_stream_stat(fp, &ssb) != -1 && (uint64_t) ssb.sb.st_size > UINT32_MAX) {
+ zend_throw_exception_ex(spl_ce_BadMethodCallException, 0, "Entry %s is too large to be stored in a phar archive", str_key);
+
+ if (save) {
+ efree(save);
+ }
+
+ if (opened) {
+ zend_string_release_ex(opened, 0);
+ }
+
+ if (temp) {
+ efree(temp);
+ }
+
+ if (close_fp) {
+ php_stream_close(fp);
+ }
+
+ return ZEND_HASH_APPLY_STOP;
+ }
+
if (!(data = phar_get_or_create_entry_data(phar_obj->archive->fname, phar_obj->archive->fname_len, str_key, str_key_len, "w+b", 0, &error, 1))) {
@@ -1678,6 +1700,7 @@ static int phar_build(zend_object_iterator *iter, void *puser) /* {{{ */
php_stream_copy_to_stream_ex(fp, p_obj->fp, PHP_STREAM_COPY_ALL, &contents_len);
if (ZEND_SIZE_T_UINT_OVFL(contents_len)) {
zend_throw_exception_ex(spl_ce_BadMethodCallException, 0, "Entry %s is too large to be stored in a phar archive", str_key);
+ data->internal_file->is_deleted = 1;
phar_entry_delref(data);
@@ -3676,6 +3699,16 @@ static void phar_add_file(phar_archive_data **pphar, zend_string *file_name, con
}
#endif
+ if (zresource) {
+ php_stream_from_zval_no_verify(contents_file, zresource);
+ }
+
+ if ((content && ZEND_SIZE_T_UINT_OVFL(ZSTR_LEN(content)))
+ || (contents_file && php_stream_stat(contents_file, &ssb) != -1 && (uint64_t) ssb.sb.st_size > UINT32_MAX)) {
+ zend_throw_exception_ex(spl_ce_BadMethodCallException, 0, "Entry %s is too large to be stored in a phar archive", filename);
+ goto finish;
+ }
+
if (!(data = phar_get_or_create_entry_data((*pphar)->fname, (*pphar)->fname_len, filename, filename_len, "w+b", 0, &error, 1))) {
@@ -3709,6 +3742,7 @@ static void phar_add_file(phar_archive_data **pphar, zend_string *file_name, con
}
if (ZEND_SIZE_T_UINT_OVFL(contents_len)) {
zend_throw_exception_ex(spl_ce_BadMethodCallException, 0, "Entry %s is too large to be stored in a phar archive", filename);
+ data->internal_file->is_deleted = 1;
phar_entry_delref(data);
goto finish;
} |
|
After that, I think it s ok but will leave it up to Wei. |
|
Thanks, I applied it in 96565a5. Before this, a failed add left an empty entry that the next write saved to the archive. Now the size check runs before the entry is created, and the test checks that no entry is left. |
LamentXU123
left a comment
There was a problem hiding this comment.
- Can you also check
Phar:mountI doubt there are similiar issues. - From what I remember tar entry's max len should be UINT32_MAX - 511 instead of UINT32_MAX? see here
| return ZEND_HASH_APPLY_KEEP; | ||
| } | ||
|
|
||
| if (php_stream_stat(fp, &ssb) != -1 && (uint64_t) ssb.sb.st_size > UINT32_MAX) { |
There was a problem hiding this comment.
I think this will wrongly reject reads when it starts from the middle and the remaining content ain't bigger than 4GB wdyt?
There was a problem hiding this comment.
You are right. I fixed it in 70b0213. The check now subtracts the current stream position from the stat size, in phar_build and in phar_add_file. The test now adds the last byte of the 4 GiB file from a seeked stream.
|
I fixed both in 70b0213.
|
LamentXU123
left a comment
There was a problem hiding this comment.
Thank you. Would you like to add a NEWS entry to document this change?
|
Thanks for the review! Maintainers asked me before not to add NEWS entries myself, so I left it out. Happy for whoever merges to add one. |
|
Okay I'd add it. One more thing: You allows entries up to UINT32_MAX, but ZIP archives also store the central directory offset in a 32-bit field right? A UINT32_MAX-byte entry pushes that offset past the limit once the local header is included and I think phar_zip.c still writes it without checking for overflow. |
|
Good catch, fixed in 142c1d1. The zip flush now fails when the central directory offset does not fit in 32 bits, so it no longer writes a wrapped value. This also covers several smaller entries that add up past 4 GiB, which was broken before this PR too. I added a zip case to the test. |
|
Uh not quite. The new guard checks the ZIP file-data offse but the central-directory size is still assigned to uint32_t cdir_size without an overflow check. |
|
I added the central directory size check in 7df092c, next to the offset check. The size can only pass 4 GiB through very large per-entry metadata, because it is written in full as the file comment. Before this change the |
|
I've taken a closer look here. Could you add these two tests? --TEST--
GH-25053: Phar size checks use filtered content rather than the underlying file size
--EXTENSIONS--
phar
zlib
--SKIPIF--
<?php
if (!getenv('RUN_RESOURCE_HEAVY_TESTS')) die('skip resource-heavy test');
if (PHP_INT_SIZE != 8) die('skip this test is for 64bit platforms only');
if (getenv('SKIP_SLOW_TESTS')) die('skip slow tests excluded by request');
?>
--ENV--
TEST_TIMEOUT=600
--INI--
phar.readonly=0
--FILE--
<?php
$src = __DIR__ . '/gh25053_filtered_stream.bin';
$size = 2**32 + 1;
$fp = fopen($src, 'w+b');
ftruncate($fp, $size);
fclose($fp);
foreach (['buildFromIterator', 'addFile'] as $method) {
echo $method, "\n";
$path = __DIR__ . '/gh25053_filtered_stream-' . $method . '.tar';
$phar = new PharData($path);
try {
if ($method === 'buildFromIterator') {
$fp = fopen($src, 'rb');
stream_filter_append($fp, 'zlib.deflate', STREAM_FILTER_READ);
try {
$phar->buildFromIterator(new ArrayIterator(['data' => $fp]));
} finally {
fclose($fp);
}
} else {
$phar->addFile('php://filter/read=zlib.deflate/resource=' . $src, 'data');
}
// Decode in chunks so the round trip does not allocate a 4 GiB string.
$fp = fopen('phar://' . $path . '/data', 'rb');
stream_filter_append($fp, 'zlib.inflate', STREAM_FILTER_READ);
$length = 0;
$allZero = true;
while (!feof($fp)) {
$chunk = fread($fp, 1024 * 1024);
$length += strlen($chunk);
$allZero = $allZero && strspn($chunk, "\0") === strlen($chunk);
}
fclose($fp);
var_dump($length === $size && $allZero);
} catch (BadMethodCallException $e) {
echo $e->getMessage(), "\n";
}
}
?>
--CLEAN--
<?php
@unlink(__DIR__ . '/gh25053_filtered_stream.bin');
@unlink(__DIR__ . '/gh25053_filtered_stream-buildFromIterator.tar');
@unlink(__DIR__ . '/gh25053_filtered_stream-addFile.tar');
?>
--EXPECT--
buildFromIterator
bool(true)
addFile
bool(true) |
|
I added both tests, and both failed on the old code. The early size check read the file size on disk, so it rejected filtered streams. 20c979e skips that check for streams with read filters, and the check after the copy still catches real overflows. compress() also cut the compressed size to 32 bits for phar and zip. 9ffab11 now throws a PharException in that case. Both tests pass now. |
|
Thank you. I will merge after CI is green. |
|
@lazerg Seems like FreeBSD CI don't have enough memory to run the test. Let's just skip it in that env. diff --git a/ext/phar/phar_internal.h b/ext/phar/phar_internal.h
index 58a30a52b1c..6df7968b2ae 100644
--- a/ext/phar/phar_internal.h
+++ b/ext/phar/phar_internal.h
@@ -293,6 +293,11 @@ struct _phar_archive_data {
uint32_t is_persistent:1;
};
+static inline uint32_t phar_entry_max_size(const phar_archive_data *phar)
+{
+ return phar->is_tar ? UINT32_MAX - 511 : UINT32_MAX;
+}
+
typedef struct _phar_entry_fp_info {
enum phar_fp_type fp_type;
/* offset within fp of the file contents */
diff --git a/ext/phar/phar_object.c b/ext/phar/phar_object.c
index 37390df05f2..c658915e951 100644
--- a/ext/phar/phar_object.c
+++ b/ext/phar/phar_object.c
@@ -1642,7 +1642,7 @@ static int phar_build(zend_object_iterator *iter, void *puser) /* {{{ */
/* read filters change the length, so filtered streams are only checked after the copy */
if (!fp->readfilters.head && php_stream_stat(fp, &ssb) != -1
- && (int64_t) ssb.sb.st_size - php_stream_tell(fp) > (int64_t) UINT32_MAX) {
+ && (int64_t) ssb.sb.st_size - php_stream_tell(fp) > (int64_t) phar_entry_max_size(phar_obj->archive)) {
zend_throw_exception_ex(spl_ce_BadMethodCallException, 0, "Entry %s is too large to be stored in a phar archive", str_key);
if (save) {
@@ -1664,6 +1664,12 @@ static int phar_build(zend_object_iterator *iter, void *puser) /* {{{ */
return ZEND_HASH_APPLY_STOP;
}
+ if (p_obj->fp == phar_obj->archive->ufp) {
+ php_stream_seek(p_obj->fp, 0, SEEK_END);
+ }
+
if (!(data = phar_get_or_create_entry_data(phar_obj->archive->fname, phar_obj->archive->fname_len, str_key, str_key_len, "w+b", 0, &error, 1))) {
zend_throw_exception_ex(spl_ce_BadMethodCallException, 0, "Entry %s cannot be created: %s", str_key, error);
efree(error);
@@ -1700,7 +1706,7 @@ static int phar_build(zend_object_iterator *iter, void *puser) /* {{{ */
data->internal_file->offset_abs = data->internal_file->offset = php_stream_tell(p_obj->fp);
data->fp = NULL;
php_stream_copy_to_stream_ex(fp, p_obj->fp, PHP_STREAM_COPY_ALL, &contents_len);
- if (ZEND_SIZE_T_UINT_OVFL(contents_len)) {
+ if (contents_len > phar_entry_max_size(phar_obj->archive)) {
zend_throw_exception_ex(spl_ce_BadMethodCallException, 0, "Entry %s is too large to be stored in a phar archive", str_key);
data->internal_file->is_deleted = 1;
phar_entry_delref(data);
@@ -1766,7 +1772,7 @@ static int phar_build(zend_object_iterator *iter, void *puser) /* {{{ */
PHP_METHOD(Phar, buildFromDirectory)
{
char *error;
- bool apply_reg = 0;
+ bool apply_reg = 0, pass_fp_is_new;
zval arg, arg2, iter, iteriter, regexiter;
struct _phar_t pass;
zend_string *dir, *regex = NULL;
@@ -1840,14 +1846,20 @@ PHP_METHOD(Phar, buildFromDirectory)
pass.base = dir;
pass.count = 0;
pass.ret = return_value;
- pass.fp = php_stream_fopen_tmpfile();
- if (pass.fp == NULL) {
- zval_ptr_dtor(&iteriter);
- if (apply_reg) {
- zval_ptr_dtor(®exiter);
+ pass.fp = phar_obj->archive->ufp;
+ pass_fp_is_new = pass.fp == NULL;
+ if (pass_fp_is_new) {
+ pass.fp = php_stream_fopen_tmpfile();
+ if (pass.fp == NULL) {
+ zval_ptr_dtor(&iteriter);
+ if (apply_reg) {
+ zval_ptr_dtor(®exiter);
+ }
+ zend_throw_exception_ex(phar_ce_PharException, 0, "phar \"%s\" unable to create temporary file", phar_obj->archive->fname);
+ RETURN_THROWS();
}
- zend_throw_exception_ex(phar_ce_PharException, 0, "phar \"%s\" unable to create temporary file", phar_obj->archive->fname);
- RETURN_THROWS();
+ } else {
+ php_stream_seek(pass.fp, 0, SEEK_END);
}
if (phar_obj->archive->is_persistent && FAILURE == phar_copy_on_write(&(phar_obj->archive))) {
@@ -1855,7 +1867,9 @@ PHP_METHOD(Phar, buildFromDirectory)
if (apply_reg) {
zval_ptr_dtor(®exiter);
}
- php_stream_close(pass.fp);
+ if (pass_fp_is_new) {
+ php_stream_close(pass.fp);
+ }
zend_throw_exception_ex(phar_ce_PharException, 0, "phar \"%s\" is persistent, unable to copy on write", phar_obj->archive->fname);
RETURN_THROWS();
}
@@ -1880,7 +1894,10 @@ PHP_METHOD(Phar, buildFromDirectory)
if (apply_reg) {
zval_ptr_dtor(®exiter);
}
- php_stream_close(pass.fp);
+ phar_obj->archive->ufp = pass.fp;
}
}
/* }}} */
@@ -1925,10 +1942,15 @@ PHP_METHOD(Phar, buildFromIterator)
pass.base = base;
pass.ret = return_value;
pass.count = 0;
- pass.fp = php_stream_fopen_tmpfile();
+ pass.fp = phar_obj->archive->ufp;
if (pass.fp == NULL) {
- zend_throw_exception_ex(phar_ce_PharException, 0, "phar \"%s\": unable to create temporary file", phar_obj->archive->fname);
- RETURN_THROWS();
+ pass.fp = php_stream_fopen_tmpfile();
+ if (pass.fp == NULL) {
+ zend_throw_exception_ex(phar_ce_PharException, 0, "phar \"%s\": unable to create temporary file", phar_obj->archive->fname);
+ RETURN_THROWS();
+ }
+ } else {
+ php_stream_seek(pass.fp, 0, SEEK_END);
}
if (SUCCESS == spl_iterator_apply(obj, (spl_iterator_apply_func_t) phar_build, (void *) &pass)) {
@@ -1939,7 +1961,10 @@ PHP_METHOD(Phar, buildFromIterator)
efree(error);
}
} else {
- php_stream_close(pass.fp);
+ phar_obj->archive->ufp = pass.fp;
}
}
/* }}} */
@@ -3705,9 +3730,9 @@ static void phar_add_file(phar_archive_data **pphar, zend_string *file_name, con
php_stream_from_zval_no_verify(contents_file, zresource);
}
- if ((content && ZEND_SIZE_T_UINT_OVFL(ZSTR_LEN(content)))
+ if ((content && ZSTR_LEN(content) > phar_entry_max_size(*pphar))
|| (contents_file && !contents_file->readfilters.head && php_stream_stat(contents_file, &ssb) != -1
- && (int64_t) ssb.sb.st_size - php_stream_tell(contents_file) > (int64_t) UINT32_MAX)) {
+ && (int64_t) ssb.sb.st_size - php_stream_tell(contents_file) > (int64_t) phar_entry_max_size(*pphar))) {
zend_throw_exception_ex(spl_ce_BadMethodCallException, 0, "Entry %s is too large to be stored in a phar archive", filename);
goto finish;
}
@@ -3743,7 +3768,7 @@ static void phar_add_file(phar_archive_data **pphar, zend_string *file_name, con
}
php_stream_copy_to_stream_ex(contents_file, data->fp, PHP_STREAM_COPY_ALL, &contents_len);
}
- if (ZEND_SIZE_T_UINT_OVFL(contents_len)) {
+ if (contents_len > phar_entry_max_size(*pphar)) {
zend_throw_exception_ex(spl_ce_BadMethodCallException, 0, "Entry %s is too large to be stored in a phar archive", filename);
data->internal_file->is_deleted = 1;
phar_entry_delref(data);
diff --git a/ext/phar/stream.c b/ext/phar/stream.c
index 0bc1f0f50cd..f7b594f45ff 100644
--- a/ext/phar/stream.c
+++ b/ext/phar/stream.c
@@ -447,7 +447,7 @@ static ssize_t phar_stream_write(php_stream *stream, const char *buf, size_t cou
{
phar_entry_data *data = (phar_entry_data *) stream->abstract;
- if ((uint64_t) data->position + count > UINT32_MAX) {
+ if ((uint64_t) data->position + count > phar_entry_max_size(data->phar)) {
php_error_docref(NULL, E_WARNING, "phar error: Could not write %d characters to \"%s\" in phar \"%s\", entry is too large", (int) count, data->internal_file->filename, data->phar->fname);
return -1;
}
diff --git a/ext/phar/tar.c b/ext/phar/tar.c
index 52541e62290..8827e1b469e 100644
--- a/ext/phar/tar.c
+++ b/ext/phar/tar.c
@@ -830,8 +830,8 @@ static int phar_tar_writeheaders_int(phar_entry_info *entry, void *argument) /*
phar_tar_octal(header.mode, entry->flags & PHAR_ENT_PERM_MASK, sizeof(header.mode)-1);
- /* the reader rejects sizes above UINT32_MAX - 511, as they overflow the 512 byte padding */
- if (entry->uncompressed_filesize > UINT32_MAX - 511
+ /* the reader rejects sizes above this limit, as they overflow the 512 byte padding */
+ if (entry->uncompressed_filesize > phar_entry_max_size(entry->phar)
|| FAILURE == phar_tar_octal(header.size, entry->uncompressed_filesize, sizeof(header.size)-1)) {
if (fp->error) {
spprintf(fp->error, 4096, "tar-based phar \"%s\" cannot be created, file \"%s\" is too large for tar file format", entry->phar->fname, entry->filename);
diff --git a/ext/phar/tests/gh25053.phpt b/ext/phar/tests/gh25053.phpt
index 07895e1209a..4530d4ea341 100644
--- a/ext/phar/tests/gh25053.phpt
+++ b/ext/phar/tests/gh25053.phpt
@@ -41,6 +41,47 @@
}
var_dump(isset($tar['big']));
+$tar = new PharData(__DIR__ . '/gh25053-iterator-retry.tar');
+$small = fopen('php://temp', 'w+b');
+fwrite($small, 'small');
+rewind($small);
+try {
+ $tar->buildFromIterator(new ArrayIterator(['small' => $small, 'big' => $src]));
+} catch (BadMethodCallException $e) {
+ echo $e->getMessage(), "\n";
+}
+var_dump($tar['small']->getContent());
+$next = fopen('php://temp', 'w+b');
+fwrite($next, 'next');
+rewind($next);
+$tar->buildFromIterator(new ArrayIterator(['next' => $next]));
+var_dump($tar['small']->getContent());
+var_dump($tar['next']->getContent());
+
+$tar = new PharData(__DIR__ . '/gh25053-iterator-read-retry.tar');
+$first = fopen('php://temp', 'w+b');
+fwrite($first, 'first');
+rewind($first);
+$second = fopen('php://temp', 'w+b');
+fwrite($second, 'second');
+rewind($second);
+try {
+ $tar->buildFromIterator(new ArrayIterator(['first' => $first, 'second' => $second, 'big' => $src]));
+} catch (BadMethodCallException $e) {
+ echo $e->getMessage(), "\n";
+}
+$third = fopen('php://temp', 'w+b');
+fwrite($third, 'third');
+rewind($third);
+$iterator = (function () use ($tar, $third) {
+ var_dump($tar['first']->getContent());
+ yield 'third' => $third;
+})();
+$tar->buildFromIterator($iterator);
+var_dump($tar['first']->getContent());
+var_dump($tar['second']->getContent());
+var_dump($tar['third']->getContent());
+
$phar = new Phar(__DIR__ . '/gh25053-4.phar');
$phar->addFromString('small', 'x');
var_dump(copy($src, 'phar://' . __DIR__ . '/gh25053-4.phar/big'));
@@ -62,11 +103,14 @@
fseek($fp, 102);
try {
$tar['big'] = $fp;
-} catch (PharException $e) {
+} catch (BadMethodCallException $e) {
echo $e->getMessage(), "\n";
}
+$tar->addFromString('big', 'small');
+var_dump($tar['big']->getContent());
$zip = new PharData(__DIR__ . '/gh25053-7.zip');
+$zip->addFromString('keep', 'old');
$fp = fopen($src, 'rb');
fseek($fp, 12);
try {
@@ -74,6 +118,10 @@
} catch (PharException $e) {
echo $e->getMessage(), "\n";
}
+var_dump($zip['keep']->getContent());
+$zip->addFromString('big', 'small');
+var_dump($zip['keep']->getContent());
+var_dump($zip['big']->getContent());
?>
--CLEAN--
<?php
@@ -81,6 +129,8 @@
@unlink(__DIR__ . '/gh25053-1.tar');
@unlink(__DIR__ . '/gh25053-2.tar');
@unlink(__DIR__ . '/gh25053-3.tar');
+@unlink(__DIR__ . '/gh25053-iterator-retry.tar');
+@unlink(__DIR__ . '/gh25053-iterator-read-retry.tar');
@unlink(__DIR__ . '/gh25053-4.phar');
@unlink(__DIR__ . '/gh25053-5.tar');
@unlink(__DIR__ . '/gh25053-6.tar');
@@ -93,10 +143,23 @@
bool(false)
Entry big is too large to be stored in a phar archive
bool(false)
+Entry big is too large to be stored in a phar archive
+string(5) "small"
+string(5) "small"
+string(4) "next"
+Entry big is too large to be stored in a phar archive
+string(5) "first"
+string(5) "first"
+string(6) "second"
+string(5) "third"
Warning: copy(): phar error: Could not write %d characters to "big" in phar "%sgh25053-4.phar", entry is too large in %s on line %d
bool(false)
mount failed
int(1)
-tar-based phar "%sgh25053-6.tar" cannot be created, file "big" is too large for tar file format
+Entry big is too large to be stored in a phar archive
+string(5) "small"
phar zip flush of "%sgh25053-7.zip" failed: archive is too large for zip file format
+string(3) "old"
+string(3) "old"
+string(5) "small"
diff --git a/ext/phar/tests/gh25053_compressed_size.phpt b/ext/phar/tests/gh25053_compressed_size.phpt
index 4ed5717a1f1..fd1fff27a0e 100644
--- a/ext/phar/tests/gh25053_compressed_size.phpt
+++ b/ext/phar/tests/gh25053_compressed_size.phpt
@@ -9,6 +9,7 @@
if (!getenv('RUN_RESOURCE_HEAVY_TESTS')) die('skip resource-heavy test');
if (PHP_INT_SIZE != 8) die('skip this test is for 64bit platforms only');
if (getenv('SKIP_SLOW_TESTS')) die('skip slow tests excluded by request');
+if (PHP_OS === 'FreeBSD') die('skip exceeds FreeBSD CI hard memory limit');
?>
--ENV--
TEST_TIMEOUT=600
diff --git a/ext/phar/zip.c b/ext/phar/zip.c
index e99327b7e9b..9a4287d805f 100644
--- a/ext/phar/zip.c
+++ b/ext/phar/zip.c
@@ -1475,7 +1475,18 @@ int phar_zip_flush(phar_archive_data *phar, zend_string *user_stub, bool is_defa
if (error) {
spprintf(error, 4096, "phar zip flush of \"%s\" failed: archive is too large for zip file format", phar->fname);
}
- goto notemperror;
+ php_stream_close(pass.centralfp);
+ if (phar->fp && pass.free_fp) {
+ php_stream_close(phar->fp);
+ } else if (must_close_old_file) {
+ php_stream_close(oldfile);
+ }
+ phar->fp = pass.filefp;
+ phar->is_brandnew = 0;
+ return EOF;
}
cdir_size = php_stream_tell(pass.centralfp); |
phar_entry_infokeeps entry sizes inuint32_t, andaddFile()/addFromString(),buildFromDirectory()/buildFromIterator()and writes through thephar://stream assigned the copied length to them without a check. A 4 GiB + 1 byte file was stored as a 1 byte entry with no error. These paths now throw (or fail the write) once the entry would not fit in 32 bits. The test needs a 4 GiB temp file, so it only runs withRUN_RESOURCE_HEAVY_TESTS.Fixes #25053