Arrgh.... That's a days graph.... I can pm you the stats server for my steamos, I can also give you root access to the steam machine iff you have ipv6 (for any future test case you want to try out, you know where to find me on IRC ;-) ).

Ok, less than an hour ago I turned on the steam machine using the controller.
At this moment I went to web (left shoulder), last pages (2x left trigger) and selected mix.dj
From that moment on the page "freezes", but from the terminal I can see steamwebhelper explode until it ooms. It got even more than 35G virt today, oh wait: swap+memory is about 45G. Well, it's a nice way to force everything unnecessary into swap 8-D.
I've noticed yesterday that after the OOM, the build-in browser worked.
I've now let it just crash on the first page, and I got an "Oops","The page you were viewing encountered an error and was terminated. Click here to reload".

Tomorrow I will test an internal plain text page after a reboot and see how that goes. I never figured it was the browser waiting on the webhelper, I always thought the renderer crashed immediately.

Couldn't resist: I rebooted.
Went to plain text->ok
Google (not plain text anymore)->ok
Went to mix.dj > OOM

This is a snapshot of an strace:

[pid  1637] <... epoll_wait resumed> {{EPOLLIN, {u32=28, u64=28}}}, 32, -1) = 1
[pid  1637] recvmsg(28, {msg_name(0)=NULL, msg_iov(1)=[{"\300\0\0\0\20\0\1\0\0\0\0\0\0\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT) = 192
[pid  1637] gettid()                    = 1637
[pid  1637] gettid()                    = 1637
[pid  1637] epoll_wait(23, {}, 32, 0)   = 0
[pid  1637] epoll_wait(23, {}, 32, 0)   = 0
[pid  1637] epoll_wait(23,  <unfinished ...>
[pid  1639] mmap(0x2edaafb38000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2edaafb38000
[pid  1639] madvise(0x2edaafb38000, 1048576, MADV_DONTNEED) = 0
[pid  1639] mmap(0x2edaafc38000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2edaafc38000
[pid  1639] madvise(0x2edaafc38000, 1048576, MADV_DONTNEED) = 0
[pid  1639] mmap(0x2edaafd38000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2edaafd38000
[pid  1639] madvise(0x2edaafd38000, 1048576, MADV_DONTNEED) = 0
[pid  1639] mmap(0x2edaafe38000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2edaafe38000
[pid  1639] madvise(0x2edaafe38000, 1048576, MADV_DONTNEED) = 0
[pid  1639] mmap(0x2edaaff38000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2edaaff38000
[pid  1639] mmap(0x2edab0038000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2edab0038000
[pid  1639] madvise(0x2edaaff38000, 1048576, MADV_DONTNEED) = 0

This is the only thing I see from strace, and before I noticed fd 28, I was too late.
Anyway: that's it: recvmsg, gettid (pid=1635), and then mmap some extra anonymous memory (malloc?), until swap+mem is full, then OOM.
I assume fd 28 is just some maintenance fd, like timers, epoll or anything.
The webhelper is probably doing something without bounds checking.

Curious... I just got it to trigger a second time in the same browsing session:
Early in the strace it uses futex a lot:

root@steamos:~# strace -c -f -p 1892
Process 1892 attached with 11 threads
Process 1960 attached
Process 1961 attached
Process 1962 attached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 63.51   17.934942        1506     11912      5919 futex
 27.30    7.708794         574     13424           epoll_wait
  8.23    2.324906     2324906         1         1 restart_syscall
  0.66    0.187381           1    170386           gettid
  0.16    0.046016           3     15820           mmap
  0.07    0.020270           5      4488           recvmsg
  0.05    0.013372           1     13664           madvise
  0.01    0.002813           2      1371           mprotect
  0.00    0.000134          22         6         6 stat
  0.00    0.000061          20         3           clone
  0.00    0.000046          23         2           write
  0.00    0.000028           9         3           setpriority
  0.00    0.000023           8         3           set_robust_list
  0.00    0.000011          11         1           munmap
  0.00    0.000007           2         3           prctl
  0.00    0.000003           3         1           tgkill
  0.00    0.000001           1         1           rt_sigprocmask
  0.00    0.000000           0         2           close
  0.00    0.000000           0         1           sendmsg
  0.00    0.000000           0         1           socketpair
------ ----------- ----------- --------- --------- ----------------
100.00   28.238808                231093      5926 total

And yes, the secret to the OOM is somewhere to be found on the http://www.mix.dj pages.
I recommend them if you like house, I need it to get my brain going.

Correction: plain text can lead to a dump too.
What I did notice:
First tab open -> webhelper goes beserk
Second tab open:we let the webhelper go beserk on the first page -> opens
after that it's just waiting until the first tab crashes and we have to close that tab. The second tab doesn't seem to use it somehow.

I'm having the same issue on Arch Linux.

I failed to mention that with the new steam client it seems to not gobble up memory anymore. At least not for me.
@svanheulen can you confirm that too?
Not only does it not gobble up memory anymore, but I can also use the crunchyroll beta html player \0/ (totally unrelated though) .

Yeah, it seems to be fixed now.

Closing since it was fixed with the new client.

I still have this. I'm running the steam Arch package from multilib.