The bitmaps are fetched from the internet and cached so you may just be seeing normal behavior of the cache not having an entry and not filling from the internet since you're in offline mode.
It makes my game library look :(
Confirmed.
Currently, Steam does not cache grid images, the only ones showing here are non-Steam games:

Steam should keep a permanent offline cache. It would save both users and Valve bandwidth to only send Steam Grid images once (or when the developer changes the image). This would solve the bug.
This issue a also present in Big Picture Mode. Unfortunately, It makes BPM barely usable in Offline Mode, because you can't see which game is currently selected in the library view.
The modern BPM caches icons properly in ~/.local/share/Steam/appcache/librarycache/
Closing per the last comment.
Nothing extracted yet.
When starting Steam in Offline Mode, most of the game icons are either small or absent in grid view, as you can see in the picture. Today, a few of them are normal size for some reason (Awesomenauts, Bastion, Beat Hazard), but that's not usually the case.
At first I thought it might have to do with not having those games installed, but that's not true. Amnesia and Aquaria are both installed, while Bastion is not.
The icons are always normal size again when I log on in Online Mode.
System Information:
Processor Information:
Vendor: AuthenticAMD
CPU Family: 0x10
CPU Model: 0x5
CPU Stepping: 0x3
CPU Type: 0x0
Speed: 3100 Mhz
4 logical processors
4 physical processors
HyperThreading: Unsupported
FCMOV: Supported
SSE2: Supported
SSE3: Supported
SSSE3: Unsupported
SSE4a: Supported
SSE41: Unsupported
SSE42: Unsupported
Network Information:
Network Speed:
Operating System Version:
Ubuntu 12.04.4 LTS (64 bit)
Kernel Name: Linux
Kernel Version: 3.8.0-39-generic
X Server Vendor: The X.Org Foundation
X Server Release: 11303000
X Window Manager: Compiz
Steam Runtime Version: steam-runtime-release_2014-04-15
Video Card:
Driver: ATI Technologies Inc. AMD Radeon HD 6700 Series
Sound card:
Audio device: VIA VT1708S
Memory:
RAM: 7984 Mb
Miscellaneous:
UI Language: English
LANG: en_CA.UTF-8
Microphone: Not set
Total Hard Disk Space Available: 232570 Mb
Largest Free Hard Disk Block: 114619 Mb
Installed software:
Recent Failure Reports:
Wed Jun 11 10:27:59 2014 GMT: file ''/tmp/dumps/crash_20140611052759_2.dmp'', upload no: ''Couldn't resolve host name''
Wed Jun 11 10:27:59 2014 GMT: file ''/tmp/dumps/crash_20140611052759_6.dmp'', upload no: ''Couldn't resolve host name''
Wed Jun 11 10:27:59 2014 GMT: file ''/tmp/dumps/assert_20140611052759_4.dmp'', upload no: ''Couldn't resolve host name''
Wed Jun 11 10:27:59 2014 GMT: file ''/tmp/dumps/assert_20140611052759_8.dmp'', upload no: ''Couldn't resolve host name''