Siphon: A MySQL-to-Redis Push Task for FuelPHP

Welcome!  This blog post, and the code to which it refers are still very much a work in progress. Please let me know if you have any questions about the content, or problems with the code. Thanks, Mike.

Update:  I should really note that this code is not production ready.  You’re going to take a serious performance hit if you try to dump your giant MySQL database into redis keys.  Use with care. 

In my spare time I’ve been working on a project with my nephew that is going to require some realtime data processing outside of my usual PHP/MySQL idiom.  However, a good deal of the project still operates in the fashion of your typical web application. Without going into a lot of specifics, information entered into the web interface will be offloaded into redis and processed via worker jobs running elsewhere.  Our issue was the idea of maintaining data in two separate places.

Now, I could have just written the web application to use redis exclusively, and that would probably have been fine.  But I thought it might be more interesting to see how easily I could map the relational data to keys in redis.  I’ve been using the FuelPHP framework on a number of projects recently, and I can say with complete honesty that I finally feel like I’m experiencing truly rapid application development.  I can ramp up a project in hours instead of days — it’s been a pleasure to work with.

That being said, the Model implementation in Fuel, coupled with the built-in Redis class, made this task (heh) a lot simpler to accomplish than I first imagined.

Introducing:  Siphon

FuelPHP is built with rapid application development in mind.  Generating controllers, models and even full-blown admin interfaces can be done in almost no time.  Models, when generated with Fuel’s built-in `oil` task, specify their properties in a protected, static parameter called, not surprisingly, $_properties.

Relationships, a must-have feature that I’ve found easier here than in any other framework or ORM library that I’ve used, are defined in a similar way, with static properties following an easy naming scheme – $_belongs_to, $_has_one, $_has_many, and the ever-popular but oft confusing $_many_many.  More information Fuel’s ORM layer can be found in the FuelPHP Docs.

Redis is obviously not a relational database, but I was able to find some correlations that worked, at least for my purposes.  How well they’ll work in the wild is for you to tell me.  Seriously, lemme get some feedback, because if there’s a better way to do it, I’m all ears.

Siphon works on the idea that we can use redis Hashes to store object representations, and Sets to store the relationships.  Let’s assume that we’re talking relationships based on the primary key, because that is all that’s being accounted for at the moment.

Note:  I’ve moved the Download/Install Instructions to a page all their own.

Further Reading

My work here has been heavily influenced by a number of fantastic resources that have been covering this sort of thing far longer than I.  Articles by Phil Sturgeon and Justin Carmony have been especially helpful.  If you’re interested in more about either Fuel or redis, take some time to browse the links below.

  1. Build an Admin Panel with the Fuel PHP Framework  by Phil Sturgeon
  2. MySQL, Redis and a Billion Rows – A Love Story by Justin Carmony
  3. Blazing Data with Redis by Justin Carmony
  4. An Explanation of Redis Data Types

Conclusion

So, for the first time in my (almost) 15-year career as a professional developer, I’m releasing some code I wrote into the wild.  I sincerely hope that someone else finds this helpful.  If you have any questions, comments or diatribes that you’d like to share, please feel free to do so below.  I’d love any feedback that you would like to share — reporting issues would be really helpful too.  And if you’d like to contribute, feel free to Fork Siphon on Github.  Thanks for reading!

Blog Neglect

Updates are coming, I assure you.  I’ve got blog posts to recap php|tek ’12, an article on my trials and tribulations with Apache Solr, and why I feel the need to scan the barcodes on every consumable item in my kitchen.

PLUS, I just ordered an IOIO Mint board, and have every intention of utilizing it in my attempt to take over the world, one android device at a time.  If you’d like to do the same, I’ll be posting tutorials as I go.  World domination, ahoy!

-M

php|tek12 Tutorial Day

Like I said before, this is my first Tek conference, so I really wasn’t sure what to expect.  I must say I’ve been very impressed with the caliber of talks and the knowledge of the presenters.  I also realized that presenting at a conference like Tek is definitely something I should do in the future.  My desire to become more involved in the community has certainly been reaffirmed – now I just need to begin contributing!

Workshop #1 – HTML5 & Javascript for the PHP Developer

This talk was in two parts, and I only caught the first.  The coverage of the new HTML5 tags and properties was very helpful.  I’ve looked at some information regarding the new CSS3 selectors, but Jake’s talk was very informational in terms of ‘stuff I actually give a damn about’.  There are certainly a number of UX elements that could benefit from an HTML5 upgrade.

Workshop #2 – Quality Assurance

Michelangelo van Dam gave an excellent presentation on the tools and processes involved in implementing Quality Assurance into your development workflow.  Numerous tools were mentioned, including Revision Control (SVN, Git, etc.), Continuous Integration (Jenkins, Phing, etc.), Testing (PHPUnit, PHP_Depend, PHP_CodeSniffer, etc.), and Bug Tracking (Jira, Mantis, Bugzilla, etc.).  I’ve looked at all of these tools before, but Mike’s presentation made me feel like implementing them into a workflow is a lot easier than it might seem just from reading tutorials.

The tutorial day of php|tek ’12 was definitely worth attending.  Most of the information presented was already familiar to me, but chatting with other developers that use these tools and implement “real” processes on a regular basis has reignited my desire to change my current development workflow to something that actually makes sense.  More updates tomorrow!

 

 

Justin Carmony on redis – php|tek12

Justin Carmony gives a presentation ‘Blazing Fast Data with Redis’ at php|tek12 in Chicago, IL.

Sorry for a video quality – I wasn’t planning on shooting any, but the demo project showing the generation of a rainbow table using php, redis and salt was just too cool not to share.

PS – The big number in the lower-right hand corner is the number of operations per second across a pool of redis servers – amazing!

php|tek ’12

I finally made it to a php|tek conference!

I’ve been trying to get up here for one of these for the last five years.  Fortunately, this year I was able to attend, and so far, it’s been great!  We’ve only just begun, but already I feel like I’m going to pick up a lot of excellent information.
I’ll post more soon, so check back for updates…

Achievement Badges from php|tek '12, sponsored by Twilio

Meet Sam

I was updating my website and cleaning up a few things, and suddenly I realized that this video, which I had never posted here, was exactly a year old.  So I’ll post it today, and share it again.

Remote desktop connection cannot verify the identity of the computer that you want to connect to…because it is lame.

Well, that’s an annoying error.  Here’s how you fix it (content blatantly copied from http://shebangme.blogspot.com/2010/05/remote-desktop-connection-cannot-verify.html

When trying to connect OS X to a Windows Machine via RDP then do the following:

1) Delete all files in /user/{username}/Documents/RDC Connections
2) Don’t fill in the domain text box