ErrorException - Notice: unserialize(): Error at offset [X] of [X+2] bytes in vendor/symfony/symfony/src/Symfony/Component/Config/Resource/FileResource.php line 78 after clearing dev cache #6203

gergelypolonkai · 2012-12-06T10:05:05Z

Sometime after I clear the dev cache with

php app/console cache:clear --env=dev

and then try to run some other command, like assets:install, I get an error like

  Notice: unserialize(): Error at offset [97 of 99 bytes in /var/www/html/vendor/symfony/symfony/src/Symfony/Component/Config/Resource/FileResource.php line 78

Although I have xdebug installed and enabled, it doesn't generate any backtrace.

The text was updated successfully, but these errors were encountered:

stof · 2012-12-07T21:10:58Z

If you want a backtrace, run the console with the --verbose option

gergelypolonkai · 2012-12-08T02:59:14Z

It's the same:

$ app/console cache:clear --verbose

  [ErrorException]                                                                                                                                                     
  Notice: unserialize(): Error at offset 97 of 99 bytes in /var/www/html/vendor/symfony/symfony/src/Symfony/Component/Config/Resource/FileResource.php line 78

gergelypolonkai · 2012-12-08T03:03:17Z

If I remove the cache directory, and run cache:clear again (or cache:warmup), everything goes back to normal.

gergelypolonkai · 2012-12-13T10:30:11Z

I have added a debug line to FileResource.php, so I could generate a backtrace. Here it is: https://gist.github.com/4275576

Actually, it tries to unserialize this string:

s:91:"/var/www/html/platea/app/cache/dev/jms_diextra/doctrine/EntityManager_50c9a1736b0f8.php";}i:4

which is clearly invalid at the given position.

ghost-x47 · 2012-12-14T08:47:23Z

I get the same error, but it's only if i cache:clear with debug .
If i provide --no-debug to cache:clear - error disappearing.
Oh, and seems like cache:warmup is working too.
Also i have no clue how did i get it, it just suddenly showed up.

gergelypolonkai · 2012-12-16T23:59:04Z

I have successfully captured an HTML version of the exception. It's 127kB and contains the whole backtrace. Where can I upload it?

Elendev · 2012-12-18T20:25:42Z

Hello, same bug here and I've also successfully captured an HTML version of the exception. It's also quite big, where can I upload it ?

Here is the first part :

ErrorException: Notice: unserialize() [function.unserialize]: Error at offset 136 of 138 bytes in /home/renaudot/public_html/frtrains/shop_app/2.0.0-rc6/vendor/symfony/symfony/src/Symfony/Component/Config/Resource/FileResource.php line 78
in /home/renaudot/public_html/frtrains/shop_app/2.0.0-rc6/vendor/symfony/symfony/src/Symfony/Component/Config/Resource/FileResource.php line 78
at ErrorHandler->handle('8', 'unserialize() [function.unserialize]: Error at offset 136 of 138 bytes', '/home/renaudot/public_html/frtrains/shop_app/2.0.0-rc6/vendor/symfony/symfony/src/Symfony/Component/Config/Resource/FileResource.php', '78', array('serialized' => 's:129:"/home/renaudot/public_html/frtrains/shop_app/2.0.0-rc6/app/cache/preprod/jms_diextra/doctrine/EntityManager_50d0bd7e3fd10.php";}i:4'))
at unserialize('s:129:"/home/renaudot/public_html/frtrains/shop_app/2.0.0-rc6/app/cache/preprod/jms_diextra/doctrine/EntityManager_50d0bd7e3fd10.php";}i:4') in /home/renaudot/public_html/frtrains/shop_app/2.0.0-rc6/vendor/symfony/symfony/src/Symfony/Component/Config/Resource/FileResource.php line 78
at FileResource->unserialize('s:129:"/home/renaudot/public_html/frtrains/shop_app/2.0.0-rc6/app/cache/preprod/jms_diextra/doctrine/EntityManager_50d0bd7e3fd10.php";}i:4')
at unserialize('a:552:{i:0;C:46:"Symfony\Component\Config\Resource\FileResource":141:{s:132:"/home/renaudot/public_html/frtrains/shop_app/2.0.0-rc6/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/FrameworkBundle.php";}i:1;C:46:"Symfony\Component\Config\Resource\FileResource":86:{s:78:"/home/renaudot/public_html/frtrains/shop_app/2.0.0-........

mvrhov · 2012-12-20T11:38:12Z

github's gist?

gergelypolonkai · 2012-12-20T13:33:13Z

The file is too large for that. When I pasted it, the chrome tab died.

jfsimon · 2013-03-01T09:24:50Z

There is an identifiable problem in cache:clear command.
As you run it with warmup option:

a new temporary cache directory is created
a new temporary kernel is booted
the temp cache directory is warmed up with the temp kernel
then the temp cache dir is renamed to replace the cleared cache dir

In order to make the new cache valid, files must be updated:

references to kernel class path must be replaced (from the temp to the real one) in serialized objects dumps
references to cache files path mus 8000 t be replaced (from the temp to the real one) everywhere it's found

One thing has been forgotten: reference to file paths are replaced in serialized object dumps too, without taking care of the \d+: prefix which contains the string lentgh.

The problem described above seems to be related to this.

jfsimon · 2013-03-01T17:00:06Z

@gergelypolonkai @ghost-x47 the #7235 PR fixes a related bug (may be the whole problem).
Would you like to test it?

gergelypolonkai · 2013-03-01T20:34:51Z

Unfortunately I cannot reproduce the issue at will. It just pops up sometimes...

Elendev · 2013-03-02T13:36:54Z

Same problem here...

This PR was squashed before being merged into the 2.1 branch (closes #7239). Commits ------- d1f5d25 [FrameworkBundle] Fixes invalid serialized objects in cache Discussion ---------- [FrameworkBundle] Fixes invalid serialized objects in cache | Q | A | ------------- | --- | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #6203 Fixes 2 problems: * malformed router chache filenames (matcher & dumper) * invalid cache file paths in serialized objects (.meta files)

stefanosala · 2013-03-07T08:08:58Z

I did not have this problem, but I have now after 5f2cea3

[2013-03-07 08:51:23] request.CRITICAL: ErrorException: Notice: unserialize(): Error at offset 156 of 118187 bytes in /var/lib/jenkins/workspace/acme/app/bootstrap.php.cache line 1316 (uncaught exception) at /var/lib/jenkins/workspace/ferrari-kopt/app/bootstrap.php.cache line 1316

stefanosala · 2013-03-07T08:13:52Z

This is the file that fails to be unserialized: https://gist.github.com/stewe/1fcb8e39dc657e7ec6ce

jfsimon · 2013-03-07T08:44:14Z

@stewe I'll try to fix this today

stefanosala · 2013-03-07T08:50:03Z

@jfsimon thanks, ping me if you need some help or more info about my environment.

jfsimon · 2013-03-07T09:58:53Z

@stewe would it be possible for you to quickly patch a file and test if it works?
You can join me with jabber on my email address.

uwej711 · 2013-03-07T22:30:57Z

@jfsimon there are more numbers that need to be updated (from appDevUrlMatcher.php.meta)

a:19:{i:0;C:46:"Symfony\Component\Config\Resource\FileResource":62:{s:50:"/vagrant/symfony/app/cache/dev/assetic/routing.yml";}

You need to update the 62 to 58 too since you replaced dev_new with dev. This file still contains in my case a reference to the temporary container:

i:18;C:46:"Symfony\Component\Config\Resource\FileResource":91:{s:79:"/vagrant/symfony/app/cache/dev/appDevDebugProjectContainer__51391405d3cf0__.php";}}

Not sure wether the latter creates an issue though ...

mpdude · 2013-03-08T10:38:38Z

How can I reproduce this?

jfsimon · 2013-03-08T10:50:30Z

@uwej711 thanks, I will use your example for testing.
@mpdude good question! It would be nice to reproduce this.

mpdude · 2013-03-08T11:51:49Z

If you have to update the element sizes/length not only in the very place where you change the strings but also upwards along nested/outer elements, this approach has no chance. Just think of a case where you have to update strings in two places and happen to have their common length somewhere up the tree, so you have to change that number by two times the delta in size.

Without having looked at the code at all I'd go for temporary strings of the same length as the original ones instead.

Clearly all this is a Kluge Scale Level 2 thing... :-(

fabpot · 2013-03-08T12:28:17Z

So, we have two options here:

unserialize, make our changes, serialise again;
use a string of the same size to avoid the problem in the first place.

uwej711 · 2013-03-08T13:39:17Z

I think the second option might be better although that leaves us with the task to find a TLA for dev_new ...

jfsimon · 2013-03-08T13:51:08Z

Yep, the second solution is the best because it's more simple.

For the directory name, I propose something like $01, if it exists (because 2 cache:clear have been executed in parallel) then $02 is tried and so on... The number can be filled with as 0 as needed to match directory name length.

uwej711 · 2013-03-08T13:57:39Z

Are you sure about the '$' ? Look's suspicious to me, does it work on all platforms? Maybe '_' instead?

mpdude · 2013-03-08T14:13:10Z

Probably opening a can of worms, feel free to ignore me - question: What assumptions can we make for the cache dir at all?

Currently, if the kernel.cache_dir is set to "/some/where/X", the code assumes it may/can create new dirs at /some/where at will (it moves X to X_old, creates X_new). And in this particular case we'd need to choose a single-character name for the new temp directory (the name for _old is less important).

If all the cache stuff was shifted into the kernel.cache_dir, we could make up arbitrary names. For example, if "/some/where/X" was given, we could have "current" and "new_000" to "new_FFF" as names inside that at discretion.

Additionally, we'd only require that we have write permissions for the given dir, not for the parent as well.

I smell a BC break, though.

jfsimon · 2013-03-09T11:06:13Z

@mpdude actualy, the cache directory is not manipulated, only subdirs are cloned/renamed (those directories named with environment name). You're right for the "1 char dir name" problem, temporary directories could maybe named with just a number, without prefix.

mpdude · 2013-03-09T11:31:04Z

@jfsimon Hm... In my setup (which admittedly has a tweaked directory layout) I have (for a particular environment)

public function getCacheDir() {
            return '/path/to/project/tmp/environment_name';
}

in "my" AppKernel and during the CacheClearCommand, a environment_name_new is created in that .../tmp directory. So the command assumes it may create a ..._new directory besides the cache dir, not inside it?

jfsimon · 2013-03-09T11:44:58Z

@mpdude ah, okay, my bad. I assumed /path/to/project/tmp (in your exemple) was the cache directory. It's a convention to have all env specific cache dirs in the same directory, and use this directory only for cache purpose. But you're right, it's not a requirement.

Anyway, if the cache parent directory is not writeable, the command fails before removing the cache with an exception, so it should not be a problem, what do you think?

mpdude · 2013-03-09T12:07:27Z

Right, my directory names obscured that. You should have one environment-specific directory for caching purposes only, and that is called the kernel.cache_directory.

My point was that the cache clear command works outside this directory, requiring suitable permissions on the parent of the cache dir and possibly running into naming conflicts.

If BC were not an issue, users could give a "cache base dir" we may write to. By setting kernel.cache_dir = cache_base_dir + "current", we'd stay below that base at all times and safely use new_XXX for temp dirs because we "own" that directory space.

mpdude · 2013-03-09T21:00:26Z

Trying to reproduce this... Obviously, the tweaks that are currently applied against the container (.php and .xml) are not a problem and string lengths make no difference there.

With a custom project I had at hand, only the .meta files for the Routing Matcher and Generator contained the temporary ..._new directory name (before replacements, of course). That was only a single match each - a FileResource added by the AsseticBundle for the routing.yml file AsseticBundle generates to inject the asset routes for the use_controller = true case.

In PHP's serialization format, it makes a difference whether a class implements \Serializable or not. Without going into detail, it looks as if the rewrite logic currently in place suffices as long as serialized classes do not implement \Serializable, because then the string length will only appear in one place immediately preceeding the string.

As we're only talking about .meta files right now, only Config\ResourceInterface implementations should matter and we currently only have File... and DirectoryResource there. I don't really see why those should implement \Serializable at all? It got there in cbd0c3c by @stof, maybe he remembers why?

When the Resources are \Serializable, I perform console cache:clear and then run a request, I get a PHP error

Warning: Insufficient data for unserializing - 132 required, 130 present

When the Resources are not \Serializable, it works.

Am I missing something? Can you guys please check your setups and projects and see if you find anything else of relevance in the .meta files?

@fabpot What do you think of moving/shifting the cache dir as I outlined above? If we could replace the "real" for the "_new" dir with no string length changes, we would not depend on implementation details like \Serializable and had a chance of fixing all kinds of files where the strings appear in plain/unencoded form, not only the .meta files.

Disclaimer
The entire idea of fiddling around with the serialized strings is an ugly hack, makes assumptions about how the ConfigCache is implemented right now and has a good chance of missing (i. e. not re-writing) any path references that are not in the container and .meta files but in arbitrary cache files written by "userland" code. It is bound to break again in the future so we need some clever ideas how we can compile/warm the container and surrounding stuff in the right place from the start on.

toloco · 2013-03-12T16:12:00Z

Hi guys,

I am really worried about this problem, I have 3 projects using symonfy2 (2.1 and 2.2) and all have the same problem after updating... there's any chance to solve it soon?

Regards,

jfsimon · 2013-03-12T16:25:29Z

@toloco yes, this will be fixed tomorrow, even if the fix is not the ideal one.

stof · 2013-03-12T17:31:39Z

@mpdude The resources implement Serializable because it is a requirement for them to be serializable. In a clean implementation, if should even be part of the interface but this has been reverted as it was breaking stuff in AsseticBundle where the resources are implemented in a bad way and thus cannot implement Serializable cleanly (which should be fixed)

mpdude · 2013-03-12T21:27:33Z

@stof Unlike Java's java.io.Serializable PHP's \Serializable has nothing to do with whether or not a class can be serialized. Any class can be serialized, either by doing/implementing nothing, having __sleep and/or __wakeup methods or implementing that interface. The difference is only how exactly serialization works and what gets serialized or not.

Thus, it is an implementation detail that should be left to the classes implementing ResourceInterface (in this case). Regarding the interface contract you are talking about, AFAIK there is no language-level way to guarantee that implementors of the interface make sure that Resources work well after being deserialized.

@toloco

This PR was merged into the 2.1 branch. Commits ------- f2ef6bc [FrameworkBundle] removed BC break cc3a40e [FrameworkBundle] changed temp kernel name in cache:clear 7d87ecd [FrameworkBundle] fixed cahe:clear command's warmup Discussion ---------- [FrameworkBundle] fixes cahe:clear command's warmup Solution taken is to replace the last char of the cache directory name to create a temporary cache directory, this way the temporary cache path has the same length than the real one. I tested this on several projects, in dev and prod environments. | Q | A | ------------- | --- | Bug fix? | yes | New feature? | no | BC breaks? | yes | Deprecations? | no | Tests pass? | yes | Fixed tickets | #6203 --------------------------------------------------------------------------- by jfsimon at 2013-03-13T12:32:25Z @toloco @gergelypolonkai @ghost-x47 @stewe it would be great if you could test this patch on your projects and report result! --------------------------------------------------------------------------- by toloco at 2013-03-13T12:41:47Z Im sorry but have the same... Notice: unserialize(): Error at offset 155 of 174227 bytes in /home/tolopalmer/Projects/shareandcoach/app/bootstrap.php.cache line 915 --------------------------------------------------------------------------- by jfsimon at 2013-03-13T12:45:04Z @toloco could you paste the backtrace in a gist? and maybe the concerned file? --------------------------------------------------------------------------- by stof at 2013-03-13T13:11:47Z @jfsimon You probably have the same issue with the name of the temporary kernel class --------------------------------------------------------------------------- by jfsimon at 2013-03-13T13:36:13Z @stof if you're right, it's a nightmare. It must be possible to write a parser/fixer for serialized objects, don't you think? --------------------------------------------------------------------------- by toloco at 2013-03-13T14:22:56Z Here you are the gist with the stack and the bootstrap.php.cache file https://gist.github.com/toloco/5152581 --------------------------------------------------------------------------- by mpdude at 2013-03-13T20:08:08Z @jfsimon Writing such a parser is painting yourself in the corner. Use a temp kernel class name of the same length as a quick fix. #7230 could bring a solution because we might be able to inject a different ConfigCache factory during the command that intercepts and substitutes Resources before they get written into the meta file. Not sure if that PR has a chance of being picked though. --------------------------------------------------------------------------- by toloco at 2013-03-14T08:19:58Z So guys? we are blocked with this problem, can I help you? I can provide more stacks if it's needed --------------------------------------------------------------------------- by mpdude at 2013-03-14T10:05:05Z @toloco Could you please post the /home/tolopalmer/Projects/shareandcoach/app/cache/dev/appDevUrlMatcher.php.meta file? That's the one that is broken. --------------------------------------------------------------------------- by jfsimon at 2013-03-14T10:15:20Z @mpdude you can find its content in the gist https://gist.github.com/toloco/5152581 (1st file, 6th line) --------------------------------------------------------------------------- by mpdude at 2013-03-14T10:24:55Z @toloco That file should contain a serialized set of Resources, it's not in the Gist. --------------------------------------------------------------------------- by jfsimon at 2013-03-14T10:33:12Z @mpdude it's more visible in the raw file: ttps://gist.github.com/toloco/5152581/raw/48a1a823b5c8e6ba03936a52e8dc0d0ff1888f8a/Error+ --------------------------------------------------------------------------- by jfsimon at 2013-03-14T10:33:27Z sorry: https://gist.github.com/toloco/5152581/raw/48a1a823b5c8e6ba03936a52e8dc0d0ff1888f8a/Error+ --------------------------------------------------------------------------- by toloco at 2013-03-14T10:37:09Z https://gist.github.com/toloco/5160317 here you are the appDevUrlMatcher.php and meta --------------------------------------------------------------------------- by jfsimon at 2013-03-14T10:51:46Z @toloco I applied @mpdude's solution (have a temp kernel class name of the same length than the real one). Could you test it to see if it fixes your problem? --------------------------------------------------------------------------- by mpdude at 2013-03-14T10:58:46Z @jfsimon Thanks! @toloco If Jean-François' fix does not work, please make sure that the .meta file you posted was the broken one? I was able to unserialize it without problems. --------------------------------------------------------------------------- by toloco at 2013-03-14T11:02:09Z Man!!!! you are the fucking boss it works!! --------------------------------------------------------------------------- by mpdude at 2013-03-14T11:04:30Z @jfsimon you just made someone happy. --------------------------------------------------------------------------- by jfsimon at 2013-03-14T11:12:39Z @toloco @mpdude \o/

jfsimon mentioned this issue Mar 1, 2013

[FrameworkBundle] Fixes invalid serialized objects in cache #7235

Closed

jfsimon mentioned this issue Mar 9, 2013

[FrameworkBundle] Fixes invalid serialized objects in cache #7239

Closed

jfsimon mentioned this issue Mar 13, 2013

[FrameworkBundle] fixes cahe:clear command's warmup #7360

Merged

fabpot closed this as completed Mar 14, 2013

Uh oh!

ErrorException - Notice: unserialize(): Error at offset [X] of [X+2] bytes in vendor/symfony/symfony/src/Symfony/Component/Config/Resource/FileResource.php line 78 after clearing dev cache #6203

ErrorException - Notice: unserialize(): Error at offset [X] of [X+2] bytes in vendor/symfony/symfony/src/Symfony/Component/Config/Resource/FileResource.php line 78 after clearing dev cache #6203

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!