HOME | DD

dt — How We Debug in DT
Published: 2012-09-13 19:06:49 +0000 UTC; Views: 19015; Favourites: 9; Downloads: 0
Redirect to original
Description body div#devskin1262065 .gr-body { background:#FFFFFF; font:10pt/1.4em Verdana; border-bottom:1px solid #A6B2A6; } body div#devskin1262065 i.gb { display:none; } body div#devskin1262065 a { text-decoration:none; color:#2f779e; } body div#devskin1262065 a:hover { text-decoration:underline; } body div#devskin1262065 img { max-width:100%; } body div#devskin1262065 strong { font:bold 15px/normal Verdana, Geneva, sans-serif; color:#404040; margin:0 0 -10px; display:block; } body div#devskin1262065 p { max-width:800px; width:auto!important; width:800px; margin:0; } body div#devskin1262065 code { font:normal 11px/20px Monaco, 'Courier New', Courier, monospace; color:#1b3c59; background:#f2f6ff; border:1px solid #ccddff; display:block; padding:6px 0 20px 10px; border-radius:3px; -moz-border-radius:3px; -webkit-border-radius:3px; margin:5px 0px 5px 0px; } body div#devskin1262065 div.positions { text-align:center; background:#0F3B6A; border-radius:4px; -moz-border-radius:4px; -webkit-border-radius:4px; padding:0 10px 1.3em 10px; margin:0 20px; } body div#devskin1262065 div.positions a { text-decoration:none; color:white; } body div#devskin1262065 div.positions a:hover { text-decoration:underline; }

As developers, we often have to debug complex data across several different layers: server-side PHP, Javascript, load time, server information, memcache events, how pages are routed, along with several other pieces of data.

In dt , we use an internal system called DRE (Developer Runtime Environment), which allows us to grab information about how the code we write is executed while making a minimal impact on the code itself.

What makes DRE especially awesome?
We can use it to help debug and solve problems that other users are having. To accomplish this, once we are aware that a particular user is having an issue, we can add DRE logging to this user, usually limiting the logging to the part of the site in which they are having the problem. We can then use the log to check for any errors that might show up.

For example, say a user named testera is having trouble with a journal. We can log debug data for him when he visits journal pages. Examining that log, several pieces of information are stored, many of which will give us a better idea of where the point of failure is. We might want to check privileges the user has or what roles they have in a group if that's where the journal is being posted.






We take this type of logging very seriously, so don't be , because we only use it as a last resort, typically only using it within our own personal development spaces with test data.

We also store the logs so we're able to link them to other developers (and even then, we only store those logs for a few hours).

Why is it helpful to developers?
Because we load all debug information in a single window, it is accessible either by clicking a box at the top of any page or through F2. DRE can log simple data types but it can also provide a highly readable breakdown of complex objects using nested tables.



We also use it to set up alerts. If a high-risk condition occurs on a particular page, the box used to display DRE will turn red and pulsate to make it more apparent to us that there is a problem. This could be something as simple as a failure to retrieve data or something as severe as a failure to include a file (or worse, a fatal error).

More importantly, it allows us to trace data and debug across multiple layers, including processes that are non-web-facing or for asynchronous tasks, where the data may not be immediately available at runtime.

It's not just for fixing bugs; it's also for helping us spotting slow-loading modules and queries and seeing where we can better optimize the number of queries we do need to run on any given page.

For example, on this page, we have 12 separate queries that run:



This is pretty good, given all the data that has to be loaded to make DeviantArt run.

But on a different page, we might have 52 queries, some which are very similar and loading data from the same tables. If we can find a way to cull them together, we can reduce the number of queries, which will help the site perform better. We have a way of handling this in dt , a technique called wenching (which is combining several queries into a single query--we'll get to this in a future article), but we might not be aware that something needs to be wenched until we get a composite look of everything taking place on the page. This is where DRE comes into play.

We can also use it for profiling to see when a file in particular is taking a long time to load. This might be because the code in that file has inefficiencies that need to be refactored. For example, here's a page just to show you how this might look:



Eek. Bad, right? Fortunately, this isn't something we see too much of. And when we do, we try to remedy it.

So how does it work?
This is a silly impractical example, but it'll do.

$orders = $this->get_orders();
dre_trace('orders', $orders);

If get_orders() returns an object of orders, DRE will trace that object into a table, labeled "orders". Simple enough.

"This sounds like it could be kind of messy when you have a bunch of developers all using it! If everyone is using DRE, the panel would be completely overwhelmed with junk I don't care about!"

Yep. That's why we use dre_trace() sparingly. We have several functions for DRE, one which takes a parameter to filter the message to only those who want to see it. For example, if I wanted to make sure only I saw $orders from the aforementioned example in our DRE panel:

$orders = $this->get_orders();
dre_cond_trace('fartprincess', 'orders', $orders);

At the end of the day, this utility saves developers a massive amount of time.


Like the idea of working on stuff like this? Check out the open developer positions at deviantART.

Related content
Comments: 22

20after4 [2013-10-09 22:01:06 +0000 UTC]

For those of us who don't have the privilege of using the cool %dt tools, there is an open source debugging tool that has some similarities to dre. Check out phpdebugbar.com/ , it's pretty cool, I don't miss DRE now that I've discovered debugbar.

👍: 0 ⏩: 1

banks In reply to 20after4 [2013-10-10 09:00:57 +0000 UTC]

Cool, that looks really nice Mukunda.

👍: 0 ⏩: 0

Itti [2012-10-05 20:12:38 +0000 UTC]

Nice! I always thought that if you get a rare, non-reproducable error, then there wasn't much you could do about it. It's really interesting to see that you have this option available for some of those particular "special case" bugs.

Keep up the interesting articles!

👍: 0 ⏩: 0

admx [2012-09-18 13:47:25 +0000 UTC]

I have no idea what this is but I love it

👍: 0 ⏩: 0

OrrinFox [2012-09-17 03:16:09 +0000 UTC]

Well this looks like a wonderful development platform here. I would definitely like the idea of working with something like this especially when it comes to something like server side coding. definitely a lot more complex than the simple development tools in chrome
Anyways looks like wonderful work here none the less to say.

P.s. Now i know this is something that devs arent "allowed to speak of" about future plans or anything.. but just in general when is any development going to be done on dAmn? no specific date but I was wondering what the overall status on that was going to be.

👍: 0 ⏩: 0

SaTaNiA [2012-09-14 13:23:28 +0000 UTC]

Interresting read even if I don't understand everything

👍: 0 ⏩: 0

zeruch [2012-09-14 07:18:04 +0000 UTC]

How much actual latency does this add, and is the tool fully homegrown or use any of the new libs/gems out there?

👍: 0 ⏩: 1

kemayo In reply to zeruch [2012-09-14 09:24:15 +0000 UTC]

For normal users it adds almost nothing; no-op versions of the functions are used. For developers with it enabled it can noticeably impact load times and, in particular, memory use. We actually run our developer environments with a higher memory limit than the site itself to compensate.

It's totally homegrown. We've also had it for quite a while; it's only had subtle changes since I joined dA over 4 years ago.

👍: 0 ⏩: 0

timmy64 [2012-09-14 06:07:13 +0000 UTC]

Is it sad that while I understood all that, I'm actually terrible at coding?

👍: 0 ⏩: 2

namenotrequired In reply to timmy64 [2012-09-14 16:42:35 +0000 UTC]

Just means it's well written I suppose

👍: 0 ⏩: 0

rotane In reply to timmy64 [2012-09-14 09:24:21 +0000 UTC]

Not at all, sounds just like me

👍: 0 ⏩: 1

timmy64 In reply to rotane [2012-09-14 09:27:51 +0000 UTC]

👍: 0 ⏩: 0

pestavous [2012-09-14 02:20:41 +0000 UTC]

one day, i may be able to actually understand what this article said. all i understood, is that under the hood of deviantART are collections of complex methods to handle the flood of data users generate.

it's really cool to learn what you're doing there to make this site work!

👍: 0 ⏩: 0

silverluna [2012-09-14 01:50:38 +0000 UTC]

Interesting to know about the logging, especially since, when I sent a bug notice, I then tend to avoid the part of the site that's 'broken' for me.

👍: 0 ⏩: 2

mudimba In reply to silverluna [2012-09-14 02:38:06 +0000 UTC]

As $fartprincess said, it is a last resort. When we do resort to it, it is usually coordinated with the user (for example through helpdesk). Most of the time bug reports give us enough information that we are able to replicate the problem using our own accounts or accounts that we specially create on our test machines.

👍: 0 ⏩: 0

fartprincess In reply to silverluna [2012-09-14 02:06:12 +0000 UTC]

I wouldn't worry too much about this We'll go to all lengths possible before enabling logging this way--it's very much a last resort and not something that happens very commonly at all.

👍: 0 ⏩: 0

ZCochrane [2012-09-14 01:43:09 +0000 UTC]

Very interesting!

👍: 0 ⏩: 0

vast-ginger [2012-09-14 01:24:37 +0000 UTC]

From a development perspective, is there a concurrent 'local' version that's synced with the dA live site, so that if anything gets borked on the local version, there's no danger to the live site? I know you said 'personal development spaces', but i'm curious. or is this a 'confidential secret'? lol

👍: 0 ⏩: 1

fartprincess In reply to vast-ginger [2012-09-14 01:31:48 +0000 UTC]

By "personal development spaces," I mean that all developers run a local version of DeviantART on a virtual machine, which allows us to work on code without the fear of completely destroying the site while doing so

We then sync code to Git. You can read more about our VM here: [link] -- we're hoping to write more on the whole process in the near future so keep an eye out!

👍: 0 ⏩: 1

Itti In reply to fartprincess [2012-10-05 20:16:09 +0000 UTC]

That's really interesting. I am just starting to learn more about team coding - repositories and syncs and branching commits - so it was really interesting to find out how it works in an environment that I'm familiar with from the user end.

I'd love to see the update on that process when it comes

👍: 0 ⏩: 0

electricjonny [2012-09-14 01:23:53 +0000 UTC]

Very interesting

While I don't exactly understand it all, it really sounds like you guys have a pretty good grasp on debugging and useful tools to help make this (massive) site run smoothly.

👍: 0 ⏩: 0

deviant-garde [2012-09-14 00:37:34 +0000 UTC]

Always interesting to read these kinds of articles from you guys. Also seems to explain a thing or two I've seen making third-party userscripts for myself and for other people.

👍: 0 ⏩: 0