Red dot ruby conference, 22 and 23 June 2017

August 2, 2017

Creating a great software is a simple goal, but hard to accomplish, especially when the software is popular. For ruby, the simple goal is to make the language better. Ruby is used by many developers all over the world. Many software have been written by ruby since last 25 years. A tiny bit of change would impact thousands of software, compatibility matters.

It was in Red Dot Ruby conf 2017 in Singapore and Matz was speaking about the simple goal of ruby.

But that doesn’t mean that ruby never broke compatibility. There was a huge breakage between 1.8 and 1.9 around 10 years ago. In 1.9 the virtual machine was replaced as part of performance improvement and especially it came with multi encoding support. It caused many people to stay at 1.8 for a long time and took almost 5 years to migrate completely to 1.9 after it’s release.

For python 2 to 3, it took almost 10 years to migrate, they are still continuing both of the versions. PHP had to drop support for version 6 for making a whole new language without compatibility. A similar thing happened for ECMAScript and they had to drop ES4. These are called second system syndrome.

Ruby learned from those and want to do it’s move step by step to avoid the second system syndrome. Throughout its improvement ruby is focused on some of it’s hidden rules, some of which are backward compatibility, memory usage, dependency, and maintainability. We will cover details on those topics in a separate article.

One of the goals is making ruby faster. Based on microbenchmarks ruby is obviously slower than C++ or Java, but it’s not that slow in a real application. To make it super fast at the level of 3 times faster, some drastic change is needed. JIT is a very potential candidate to achieve such goal.

There are some 3rd party libraries for JIT implementation having some pros and cons. Considering the hidden regulations 3rd party implementations might not be a good option. Ruby need to have it’s own JIT implementation, which is very difficult though. Around 1st quarter 2017 MJIT option came, which uses RTL (Register Transfer Language) and GCC to implement JIT. In the world of virtualization, there is a long argument between register based internal representation vs stack based. But at least for the purpose of JIT Register based IR is a little bit easier to implement JIT on top of it and also to optimize. They also generate less memory traffic and claim to run faster.

High level blocks of MJIT implementation | Red dot ruby conference

High-level blocks of MJIT implementation (Matz’s speech in RDRC 2017)

MJIT execution steps are:
1. Generate RTL
2. Generate C code from RTL
3. Compile C by GCC
4. Load compiled .so file
5. Execute

It has many unnecessary steps but there are a couple of tricks to optimize the process, such as using minimized header, using worker thread, ahead of time compilation. At the end it’s much faster. There are a couple of techniques yet to implement to make MJIT even faster. At current stage benchmark results are:

– MJIT runs 6x faster than Ruby2.0 on average but consumes 4x more memory than Ruby2.0.
– JRuby runs 2.5x faster than Ruby2.0 on average but consumes 200x more memory than Ruby2.0.

Optcarrot benchmark:
– MJIT runs 2.8x faster than Ruby2.0, consumes 1.16x more memory than Ruby2.0.
– JRuby runs 2.4x faster than Ruby2.0 but consumes 18x more memory than Ruby2.0.

Optcarrot benchmark result which reflects more real-life results is not that bad. But there are yet to improve. There is no promise yet if MJIT would come as the Ruby JIT engine in Ruby 3. But we have at least have an implemented technology to make ruby 3 times faster. And MJIT has the highest score towards that goal.

Ruby 3 may have more magic to come, such as multi-core aware concurrency, type inference etc. which also have to comply with the hidden regulations of ruby. Those are not yet promised too. There are tons of work to make ruby better.

Although it’s very hard to make Ruby 3×3, it is gradually moving forward towards that goal. We all hope Ruby3 is going to be a lot better than what we have now.

In the conference, Akira Matsuda showed us some recent core ruby features, how they are being used in rails and some of his works as a committer of both. He discussed mostly Module#prepend, public include, Refinements, keyword arguments, frozen string, symbol GC, private def, Multibyte encoding. Let us see some details about a couple of them here:

In pre-Module#prepend era similar feature was achieved by alias_method_chain in active support and AMC had some problem. AMC’s initial definition was like:

class Module
  def alias_method_chain(target, feature)
    alias_method "#{target}_without_#{feature}", target
    alias_method "#{target}_with_#{feature}"

which returns unexpected behavior if multiple plugins were bundled with same AMC call or there was method definition in code with the same name. It would completely depend on the load order of plugin. It’s not a clean way of implementation but simple enough.

Its usage is like:

gem 'activesupport', '4.2.8'
require 'active_support/core_ext/modele/aliasing'

class C
 def foo() p:foo; end

class C
 def foo_with_bar() foo_without_bar; p:bar; end
 alias_method_chain :foo, :bar
#=> :foo, :bar

Module#prepend is the clean way of achieving the same feature and it’s in ruby 2.0 core.
We can use it like:

class C
 def foo() p:foo; end

module M
 def foo() super; p:bar; end
C.send :prepend, M
#=> :foo, :bar

But there is a pitfall when both AMC and Module#prepend mixed together. Loading AMC first and then M#prepend works, but the other load order falls in an infinite loop. So, they can never coexist. So, AMC is terminated in rails 5.

Keyword arguments were introduced in ruby 2.0, which makes the method calling behavior more strict. It also makes the method definition as a great documentation. Akira made a branch for experimenting kwargs in before ruby 2.0.0 stable release and found some issues:

E.g. a keyword argumented method can be defined as

def x(if: nil); end

and called like

x if: true

It works fine. But we could never access the if as a variable inside the method. E.g. if the method definition would be like

def x(if: nil); p if; end

And called like:

x if: true 
#=>syntax error, unexpected keyword_end

Which is valid, “if = 1” is not a valid ruby code, so it is not possible. Kwargs opened a new possibility, which is binding.local_variable_get.

def x(if: nil); p binding.local_variable_get :if; end 
x if: true 
# true

It does the job in such case.

Akira’s proposal is to make kwargs as the first class citizen in rails and he is going to work for that. But keyword argument is still slow in ruby <2.2, because it uses ruby hash internally. It’s a bit better in 2.3 and 2.4.

private def syntax came to a recent version of ruby 2, a tiny improvement in method visibility. Earlier it was like:

class Klass
 # Changes visibility of the methods defined below
 def private_method; end


def private_method; end
# Changes visibility of a single method
private :private_method

Now it’s possible to write the syntax below too, which is like C++ or Java:

private def private_method; end

There were as some changes in the behavior of method definition, for this new syntax. def statement used to return nil in an earlier version, but its implementation was changed to return the method name in symbol now. private takes a symbol and changes the method’s visibility.

But private def is very rarely used. One or the reason could be, there was a documentation problem. Rdoc could not generate documentation properly if the method is defined using private def until recent version. Rdoc skipped any other method defined below private def, like the code below:

class Klass
 def public_method_1; end
 private def private_method; end
 # This method is a public method. But RDoc was treating this as private method because of 
the private def definition on top of it.
 def public_method_2; end

Rdoc has its own ruby parser for generating the documentation. Replacing the parser could solve the problem though. But Akira implemented a patch for fixing that without replacing the parser, and that’s parts of rails now.

In an upcoming version of rails, multibyte encoding support might be dropped from active_support. From Ruby 1.9+ multibyte encoding is supported. In reality, people are not using this feature much even in Japan. Almost everyone uses only UTF-8. Also in recent versions of Ruby lots of improvements happened in encoding handling. Those improvements are good enough to drop multibyte encoding support from active_support. If necessary developers can use ruby’s native implementation of multibyte encoding.

Ruby features are getting improved a lot in its recent versions 2.3, 2.4, 2.5. These features cannot yet be used in rails codebase yet, as rails master still supports 2.2. But we can use them in our apps. It is recommended to update production ruby version to newest stable and start using those features. No reason to stay on an old version of ruby.

Florian Weingarten’s presentation about how Shopify manages their huge ruby development was really hard to digest. Shopify platform has more than 400k shops on top of it having 20-40k requests per second. It is one of the oldest and largest codebases in ruby. It has ~800 contributors and everybody can merge in master (insane dangerous!!). That’s how they can ship 40-50 deploys per day. Another option could be to allow the only couple of people to allow for master and do bigger deploys infrequently, which would be a super insane job.

Shopify uses more deploys in short time. But, if the deploy becomes slower than ~10 min it becomes a productivity problem. They took every necessary thing to do to make the deployment at speed.

To overcome speed they use parallel CI build. Another approach is to build containers in advance and quickly, which means all the containers don’t always go to production. The advantage is if someone wants to ship a container it’s already in their hand, which avoids build time on the go. Another important factor is to educate all about better collaboration, writing better code and better error message. And the last one is tracking and alerting unreliable tests and automate them.

Daniel Bovensiepen’s talk about developing a wheelchair driver with ruby was quite exceptional. It covered his works about developing a ruby software in a raspberry pi based system.

Trailblazer creator Nick Sutterer’s keynote was also very inspiring. Along with showing some reason about why ruby is not going to be dead, he also showed about his trailblazer framework. The video is available here.

Aaron Patterson’s speech was about ruby’s incremental improvements on garbage collection. He looks completely different in the real world than the internet. It was very much enjoyable and was set as the last speech of the conference, the video is available here.

There were lots of discussions about different features of ruby, development practices, refactoring and different new Ruby-based frameworks. It’s very difficult to cover the summary of all the topics. I recommend to take a look at the conference videos and just hold tight with Ruby and Ruby frameworks.

Contributor: Fattahul AlamNascenia

No comments

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.