Tips and Resources For Setting Up Website From Ground Up

Mon, 14 Sep 2015 11:37:37 -0400

This is a summary of the whole procedure of How to build your own website using Rails Framework. It sounds really simple in the Rails Guide, but if really want to go over the whole procedure, there are steps beyond writing a small App that runs on your own computer. It is kind of a summary that reminds me things I need to care about when running a serious App in production environment, so I think it suits people who just learn how to build Rails App while want to know more about how to actually run it in production environment. This Topic covers all this contents below:

Simple Rails App & Github
Domain, DNS & Server
Deployment
Web Server & App Server

In the future there might have:

Database
Cache & Queue
Security

But for now I’m only focusing on getting through the most basic steps.

1.Simple Rails App & Github

If you already familiar with Rails and Github, just ignore this first step.

Build a Toy Rails App is probably the most easy part of this article, just follow the detailed Getting Started with Rails article here. To follow up, please go over it and I’m assuming you now have a code repository called blog on your own computer, and with command line rails s you can run it and view it in browser with address localhost:3000.

If you consider Rails too fancy for you and you just care about building static website (e.g.: for blog), use Jekyll is also a nice choice to get started, with this guide you can setup a toy blog repository in seconds, and view in with localhost:4000.

To set up production environment, you also need Version Control, nowadays most web developers use Git, you can use git with Github, or you can set your own Gitlab Web App to manage your code, or else you can just simply use git on server solely with command line, its your choice to use which method to apply. Though this article mainly focus on using Git with Github. There is also a very nice written procedure on Github here, after following these guidelines you should have a github repository called blog now.

2.Domain, DNS & Server

Many companies provides their own whole package of App management service, like Amazon AWS, Heroku etc, and to be more advanced, people also invent a container called Docker. They are pretty good if you want to develop/copy a prototype really fast. But to some extent, for beginners, they are too much and might freaks you out with their loads of services. So let’s separate them to several basic pieces first.

Domain

To make your project online, you need a domain, which is simply an address you have on the Internet. You can easily get a third level domain for free if you just want to publish a blog(e.g.: myblog.tumblr.com), but to get your own second level domain, you probably need to rent one. (Let me know if you can find a free one. :) ) Personally I use domain providers like GoDaddy and Gandi, and I prefer Gandi more because their Interface is cleaner. To get a domain, simply go to one of this providers and rent one name you like, it won’t be too expensive for the first year. After the payment you should get a domain name like foobar.com

Server

Most developers don’t use physical servers now, I suggest renting a simple virtual host, it is convenient because guys in charge of the real servers will help you fix security issues and upgrade your hardware without you even notice. Then all you need to care about is on the software side, besides, real servers are really expensive and too fancy to play with. I personally use Digital Ocean, they use SSD hard drives and the price of minimum setting is cheap. Another popular choice is Linode, I think they have a longer history and some little birds of mine said that the setting of Linode server is more comprehensive. After the payment and very easy settings (If you use Digital Ocean also), you should have a Server System installed, let’s assume it is the latest stable version of Ubuntu, and in order to make your App work, you also need set up the environment for your App to run, with sudo permission it shouldn’t be hard. And the most important thing is that, now you get your own server IP address like 123.132.213.231

DNS

DNS, according to Wikipedia, is short for Domain Name System, but to understand it in the most superficial way, it is the thing that connect your domain name and your IP address.

There are two awesome and practical articles that can lead you through the procedure of making this connection.

As you can observe, there are lots of good articles about server configuration on DigitalOcean, another reason I use it. After this step, you should able to ping your server with the domain name like the second article said.

3.Deployment

Most of times I use Capistrano, the thing about Capistrano is that there is some significant difference between Capistrano 2 and Capistrano 3, to keep up with the pace, I’ll look into details of Capistrano 3 here. FYI, there is also a deploy tool called Mina that is getting popular.

After install Capistrano via the Official Guide, there are three files you need to change. I’ve detailed comment the configuration usage in the code below.

/Capfile

# Load DSL and Setup Up Stages
require 'capistrano/setup'

# Includes default deployment tasks
require 'capistrano/deploy'

# Includes tasks from other gems included in your Gemfile
#
# For documentation on these, see for example:
#
#   https://github.com/capistrano/rvm
#   https://github.com/capistrano/rbenv
#   https://github.com/capistrano/chruby
#   https://github.com/capistrano/bundler
#   https://github.com/capistrano/rails
#
# require 'capistrano/rvm'
# require 'capistrano/rbenv'
# require 'capistrano/chruby'

# Add bundler support
require 'capistrano/bundler'
# Add assets precompile support
require 'capistrano/rails/assets'
# Add database migration support
require 'capistrano/rails/migrations'
# Add unicorn support
require 'capistrano3/unicorn'

# Loads custom tasks from `lib/capistrano/tasks' if you have any defined.
Dir.glob('lib/capistrano/tasks/*.rake').each { |r| import r }

/deploy.rb

# config valid only for Capistrano 3.4
lock '3.4.0'

# set the directory the code deployed to on server
# # Default deploy_to directory is /var/www/my_app
set :deploy_to, '/your_project'

# set your user name on server
set :user, "username"

# set the application name
set :application, 'app_name'

# set the repository address
set :repo_url, 'git@github.com:app_name.git'

# set the deployed branch
# Default branch is :master
# ask :branch, proc { `git rev-parse --abbrev-ref HEAD`.chomp }.call
set :branch, "your_branch"

# files we want sym linking to specific entries in shared.
# Default value for :linked_files is []
set :linked_files, %w{config/database.yml config/secrets.yml}

# dirs we want symlinking to shared
# Default value for linked_dirs is []
set :linked_dirs, %w{log tmp/pids tmp/cache tmp/sockets vendor/bundle public/system lib/data}

# Set application environment
set :rails_env, "production"

# Set deploy method
set :deploy_via, :copy

#########################################
# Set this configurations if you don't have sudo permission

# Deploy without using sudo if you are not sudo
set :use_sudo, false

# set tmp dir to your home directory(when you have no sudo permission)
set :tmp_dir, "/home/users/your_username/tmp"

# Default value for default_env is {}
# If you installed ruby locally(in your home directory, you need to specify its location to Capistrano)
set :default_env, { path: "/home/users/your_username/.rvm/gems/ruby-2.1.5/wrappers:$PATH" }

#########################################
# These are default settings that you probably don't need to change

# set software configuration management tool
# Default value for :scm is :git
# set :scm, :git

# Deploy console log format setting
# Default value for :format is :pretty
# set :format, :pretty

# Default value for :log_level is :debug
# set :log_level, :debug

# indicate use of sudo without password
# Default value for :pty is false
# set :pty, false

# Set the number of releases you want to keep
# Default value for keep_releases is 5
# set :keep_releases, 5

# Unicorn settings(We'll talk about it soon, just showcase the typical usage here)
set :unicorn_pid, -> { "#{shared_path}/tmp/pids/unicorn.pid" }
set :unicorn_config_path, "config/unicorn.rb"
set :unicorn_rack_env, 'deployment' # "development", "deployment", or "none"

namespace :deploy do

  desc 'Restart application'
  task :restart do
    on roles(:app), in: :sequence, wait: 5 do
      invoke 'unicorn:legacy_restart'
    end
  end

  after :publishing, 'deploy:restart'
  after :finishing,  'deploy:cleanup'
end

# After deploy cleanup, clean up old version dependencies.
namespace :bundle do
  task :clean do
    on roles(:web) do
      within release_path do
        with rails_env: fetch(:rails_env) do
          execute :bundle, :clean
        end
      end
    end
  end
end

after "deploy:cleanup", "bundle:clean"

/config/deploy/production.rb

# set deploy stage to production
set :stage, :production

# add your domain name and username on server
server 'mydomain.com', roles: %w{web app db}, user: 'username'

Then you’re all set on production deploy configuration.

4.Web Server & App Server

The easiest way is to use Passenger, the typical way is to use Nginx+Unicorn, besides Passenger and Unicorn, Puma is also a popular app server choice. Here I would describe how to set up Nginx+Unicorn on server.

First one need to install Nginx on server and unicorn gem, notice here that unicorn is a gem that should be part of the dependency of your Rails App, while Nginx is a popular web server that can work with lots of frameworks beyond Rails. After installation, there are three files that is important for configuration, I’ve commented the important part in the code below

Some of comments originally from other people’s scripts, the references are listed below. Thanks if any of you see this. :)

References:

How to Configure Nginx

Setting Up Your Production Server With Nginx and Unicorn

Capistrano3 を使って Rails4 + unicorn + nginx + rbenv にデプロイする

unicorn.rb

# Sample verbose configuration file for Unicorn (not Rack)
#
# This configuration file documents many features of Unicorn
# that may not be needed for some applications. See
# http://unicorn.bogomips.org/examples/unicorn.conf.minimal.rb
# for a much simpler configuration file.
#
# See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete
# documentation.

# app path in production environment
app_path = File.dirname(File.dirname(Dir.pwd))

# Use at least one worker per core if you're on a dedicated server,
# more will usually help for _short_ waits on databases/caches.
# But to be honest, for a small app 2 process is enough.
worker_processes (ENV['RAILS_ENV'] == 'production' ? 2 : 1)

# Since Unicorn is never exposed to outside clients, it does not need to
# run on the standard HTTP port (80), there is no reason to start Unicorn
# as root unless it's from system init scripts.
# If running the master process as root and the workers as an unprivileged
# user, do this to switch euid/egid in the workers (also chowns logs):
# user "unprivileged_user", "unprivileged_group"

# Help ensure your application will always spawn in the symlinked
# "current" directory that Capistrano sets up.
# working_directory "/path/to/app/current"
# available in 0.94.0+
working_directory "#{app_path}/current"

# listen on both a Unix domain socket and a TCP port,
# we use a shorter backlog for quicker failover when busy
listen app_path + '/shared/tmp/sockets/unicorn.sock', backlog: 64

# For development, you may want to listen on port 3000 so that
# you can make sure your unicorn.rb file is soundly configured.
listen(3000, backlog: 64) if ENV['RAILS_ENV'] == 'development'

# default timeout is 60 seconds
timeout 60

# Set the location of the unicorn pid file. This should match what we put in the
# unicorn init script later.
pid app_path + '/shared/tmp/pids/unicorn.pid'

# You should define your stderr and stdout here. If you don't, stderr defaults
# to /dev/null and you'll lose any error logging when in daemon mode.
stderr_path app_path + '/shared/log/unicorn.stderr.log'
stdout_path app_path + '/shared/log/unicorn.stdout.log'

# combine Ruby 2.0.0 dev or REE with "preload_app true" for memory savings
# http://rubyenterpriseedition.com/faq.html#adapt_apps_for_cow
preload_app true
GC.respond_to?(:copy_on_write_friendly=) and
  GC.copy_on_write_friendly = true

# Enable this flag to have unicorn test client connections by writing the
# beginning of the HTTP headers before calling the application.  This
# prevents calling the application for connections that have disconnected
# while queued.  This is only guaranteed to detect clients on the same
# host unicorn runs on, and unlikely to detect disconnects even on a
# fast LAN.
check_client_connection false

# local variable to guard against running a hook multiple times
run_once = true

before_fork do |server, worker|
  # the following is highly recomended for Rails + "preload_app true"
  # as there's no need for the master process to hold a connection
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!

  # Occasionally, it may be necessary to run non-idempotent code in the
  # master before forking.  Keep in mind the above disconnect! example
  # is idempotent and does not need a guard.
  if run_once
    # do_something_once_here ...
    run_once = false # prevent from firing again
  end

  # The following is only recommended for memory/DB-constrained
  # installations.  It is not needed if your system can house
  # twice as many worker_processes as you have configured.
  #
  # This allows a new master process to incrementally
  # phase out the old master process with SIGTTOU to avoid a
  # thundering herd (especially in the "preload_app false" case)
  # when doing a transparent upgrade.  The last worker spawned
  # will then kill off the old master process with a SIGQUIT.

  # old_pid = "#{server.config[:pid]}.oldbin"
  # if old_pid != server.pid
  #   begin
  #     sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
  #     Process.kill(sig, File.read(old_pid).to_i)
  #   rescue Errno::ENOENT, Errno::ESRCH
  #   end
  # end


  # Throttle the master from forking too quickly by sleeping.  Due
  # to the implementation of standard Unix signal handlers, this
  # helps (but does not completely) prevent identical, repeated signals
  # from being lost when the receiving process is busy.
  # sleep 1

end

after_fork do |server, worker|
  # per-process listener ports for debugging/admin/migrations
  # addr = "127.0.0.1:#{9293 + worker.nr}"
  # server.listen(addr, :tries => -1, :delay => 5, :tcp_nopush => true)

  # the following is *required* for Rails + "preload_app true",
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection

  # if preload_app is true, then you may also want to check and
  # restart any other shared sockets/descriptors such as Memcached,
  # and Redis.  TokyoCabinet file handles are safe to reuse
  # between any number of forked children (assuming your kernel
  # correctly implements pread()/pwrite() system calls)
end

nginx.conf

# This goes to /etc/nginx/nginx.conf (or where nginx is installed)

user www-data;
worker_processes 2;
pid /var/run/nginx.pid;

events {
  worker_connections 768;
}

http {

  ##
  # Basic Settings
  ##

  sendfile on;
  tcp_nopush on;
  tcp_nodelay off;
  keepalive_timeout 65;
  types_hash_max_size 2048;
  server_tokens off;

  # server_names_hash_bucket_size 64;
  # server_name_in_redirect off;

  include /etc/nginx/mime.types;
  default_type application/octet-stream;

  ##
  # Logging Settings
  ##

  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;

  ##
  # Gzip Settings
  ##

  gzip on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied any;
  gzip_min_length 500;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

  ##
  # Unicorn Rails
  ##

  upstream unicorn {
    server unix:/your_project/current/tmp/sockets/unicorn.sock fail_timeout=0;
  }

  ##
  # Virtual Host Configs
  ##
  include /etc/nginx/sites-enabled/*;

}

default

# This file goes to /etc/nginx/sites-available/default
# Then create a soft symlink of this file in /etc/nginx/sites-enabled/default

server {
  listen 80;
  server_name your_domain.name;
  root /your_project/current/public;

  client_max_body_size 1G;

  keepalive_timeout 300;

  try_files $uri/index.html $uri.html $uri @unicorn;

  location @unicorn {
    proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forward_Proto $scheme;
    proxy_redirect off;
    # This passes requests to unicorn, as defined in /etc/nginx/nginx.conf
    proxy_pass http://unicorn;
    proxy_read_timeout 300s;
    proxy_send_timeout 300s;
  }

  # You can override error pages by redirecting the requests to a file in your
  # application's public folder, if you so desire:

  error_page 500 502 503 504 /500.html;
  location = /500.html {
    root /your_project/current/public;
  }

  error_page 404 /404.html;
    location = /404.html {
    root /your_project/current/public;
  }
}

There are lots of instructions of the server configuration procedure because servers run in various environments, the code above showcase the most typical configuration from my perspective.

After complete all these configurations, restart nginx server and deploy your code again, then your site should properly running now.

How to Install CRF++ on Server Without Sudo Permisson

Thu, 06 Aug 2015 13:00:28 -0400

CRF++ is a API that implements Conditional random field technique, it also has a Python API. It is used in one of my projects and I need to install in on server without sudo permission, the author didn’t specify how to do that in detail. So I think it might be helpful to share my experience, here is how to do this:

1. Install Python in home directory

With sudo permission one can easily install Python using one line command, but without that, Python can be installed to, just a little bit trickier, first download the specific version of Python under Python Source Releases, here I chose Python 2.7.10:

wget https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tar.xz
tar -xvf Python-2.7.10.tar.xz
cd Python-2.7.10

There is a configure script under the unzipped directory and it helps generate make files and check dependencies. If execute:

./configure --help

There will be information like this:

Installation directories:
  --prefix=PREFIX         install architecture-independent files in PREFIX
                          [/usr/local]
  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
                          [PREFIX]

By default, 'make install' will install all the files in
'/usr/local/bin', '/usr/local/lib' etc.  You can specify
an installation prefix other than '/usr/local' using '--prefix',
for instance '--prefix=$HOME'.

Because without sudo permission one usually can’t write to /usr/local/ dir, so here I changed the prefix, notice you need to mkdir yourself, configure won’t do that for you:

./configure --prefix=$HOME/local/python

Then compile python with make files generated:

make
make install

After the installation, check if success:

$HOME/local/python/bin/python --version
>> Python 2.7.10

If want to set it as the default python version, one need to change the import path in profile(my default profile is .bash_profile) by add line below:

export PATH="$HOME/local/bin/python:$PATH"

Then check if success:

source .bash_profile
python --version
>> Python 2.7.10
which python
>> /your_home_dir/local/python/bin/python

2. Install CRF++

First download CRF++ source file from here, I’m using version 0.58. Then go into the unzipped file, install with prefix just as described above:

./configure --prefix=$HOME/local/CRF++
make
make install

$HOME/local/CRF++/bin is where command crf_learn and crf_test in, so don’t forget to add local CRF++ bin path in profile file:

export PATH="$HOME/local/CRF++/bin:$PATH"

Then test if CRF++ is correctly installed:

source .bash_profile
which crf_learn
>> /your_home_dir/local/CRF++/bin/crf_learn

3. Install CRF++ python API

After both Python and CRF++ is successfully installed, now we can install the python API, first cd in the python dir under CRF++ install package, and there are files like these:

CRFPP.py
CRFPP_wrap.cxx
README
setup.py
test.py

After reading the file setup.py script, apparently the auther is using distutils.core.Extension to add the C-Extension files in python, the original code is like this:

#!/usr/bin/env python
from distutils.core import setup,Extension,os
import string

setup(name = "mecab-python",
      py_modules=["CRFPP"],
      ext_modules = [Extension("_CRFPP",
                               ["CRFPP_wrap.cxx",],
                               libraries=["crfpp", "pthread"])
                     ])

However from the document Building C and C++ Extensions with distutils, the default dirs for this setup to find C include files and lib files are /usr/local/include and /usr/local/lib so we need to make change the setup.py file like this:

#!/usr/bin/env python

from distutils.core import setup,Extension,os
import string

setup(name = "mecab-python",
      py_modules=["CRFPP"],
      ext_modules = [Extension("_CRFPP",
                               ["CRFPP_wrap.cxx",],
                               include_dirs = ['/your_home_dir/local/CRF++/include'],
                               library_dirs = ['/your_home_dir/local/CRF++/lib'],
                               libraries=["crfpp", "pthread"])
                     ])

Then we can run in the directory:

python setup.py build
python setup.py install

Then for the final step, we need to add our local CRF++ lib to the system shared path so Python can find it, the detailed explanation is in here, but basically, we just need to add this line to our profile:

export LD_LIBRARY_PATH="$HOME/local/CRF++/lib:$LD_LIBRARY_PATH"

Then again:

source .bash_profile

To test if the python API is successfully installed:

python
>> import CRFPP
>> CRFPP
<module 'CRFPP' from '/your_home_dir/local/python/lib/python2.7/site-packages/CRFPP.pyc'

That’s it, then you can just explore the usage of CRFPP happily. :)

Sublime Text Tips

Sun, 22 Jun 2014 00:00:00 -0400

I’ve been use Sublime Text on my Mac for 2 years now and still it has many unknown features that would surprise me now and then. However, sometimes I do forgot about features, so I write this article as a reminder of future configuration and usage guide.

Basic Setting

Normal Setting of Sublime Text is done in Preference/Setting part, notice that usually one only need to adapt User setting and keep Default setting as it is.

My user setting is like this:

{
    "folder_exclude_patterns":
    [
        "log",
        ".git",
        "cache"
    ],
    // This three line make my code reading faster.
    "font_size": 16,
    "highlight_line": true,
    "indent_to_bracket": true,
    // I'm not a very good typer so I need this, but some people may find it annoying since it keeps telling you your perfect variable name is wrongly spelled. :)
    "spell_check": true,
    // This last three is very helpful if you really care about clean code.
    "translate_tabs_to_spaces": true,
    "trim_trailing_white_space_on_save": true,
    "ensure_newline_at_eof_on_save": true
}

Setup command line

Next step is to open files and directories from command line using Sublime Text, the installation didn’t do it for you so you need to do it by yourself, luckily it is not hard at all. :)

According to the documentation here

What you need to do is simply:

ln -s "/Applications/Sublime Text 2.app/Contents/SharedSupport/bin/subl" ~/bin/subl

But you also need be careful about your bash profile configuration, make sure to add /usr/local/bin in your export PATH and if you’re using zsh like me, make sure to change the configuration of zsh as well.

I don’t suggest to adapt default editor in terminal to Sublime Text though, when I used it as default editor there is some saving issues I encountered.

Package Control

Package Control is another amazing thing about sublime text you can always find the package you want here, several packages I used most often is as below:

Abacus It’s small and it’s old, but it is useful enough for alignment.

Markdown Preview I used it a lot, it can build github-markdown-like pages.

DashDoc Dash + Sublime Text

Search

Searching in Sublime Text is amazing, actually that’s why I fall in love with it in the first place.

Search current page is basically accomplished by Command+F, You can also use Command+P, where you input @ sign to search for function and classes and input : sign to search line number, and input # to search anything. But I mostly use Command+P to search for file names in current directory, it has super good fuzzy search support so basically you can type anything in your head and it will help you open the file you want.

To search words in files, the command you need is Command+Shift+F, notice that it also support regular expression search by press Alt+Command+R (after press Command+Shift+F), case sensitive search by press Alt+Command+C.

You can also use fuzzy search on file locations too, you can put /dir/, *.csv as your search location and then it will only do search in csv files.

Another trivial tip

To bulid your code(I’m refering to simple snippit here) from the editor, simply push Command+B, you can also write your own build method if you like. I write my own tcl build method (What is tcl?) for instance.

What’s next

I learned most of the tricks in Sublime Text Unofficial Documentation, hope to learn more as time goes by.

Learning Compiler -- Semantic Analysis, CodeGen and Optimization

Wed, 21 May 2014 18:36:00 -0400

This is the summary of Stanford Compiler Lessons on Coursera

Semantic Analysis

Semantic Analysis is used to understand the meaning of code, this is the hardest part for me.

Analyse Scope

There are two kinds of scopes, static scope and dynamic scope, most languages has static scopes

In the example language Cool, there are 5 kinds of scopes:
- Class declarations (introduce class names)
- Method definitions (introduce method names)
- Let expressions (introduce object id’s)
- Formal parameters (introduce object id’s)
- Attribute definitions (introduce object id’s)
- Case expressions (introduce object id’s)
also, methods and class names have complex scopes.
Symbol Table

A data structure that tracks the current bindings of identifiers, we can use a stack to represent it. we can first, build an abstract syntax tree, then use symbol table to check and bind things together, so if something is not right, we claim an error.
Types

A language’s type system specifies which operations are valid for which types. The goal of type checking is to ensure that operations are used with the correct types.
- The types in Cool are:
  - Class Names
  - SELF_TYPE
- How to configure type:
  - The user declares types for identifiers
  - The compiler infers types for expressions
Type Checking

using logical rules to infer the type of one expression.

Building logical blocks:
- Symbol ^ is “and”
- Symbol => is “if-then”
- x:T is “x has type T”
- |- means “it is provable that…”
Notice: Types are computed in a bottom-up pass over the AST
Type Environment

Since variables don’t have a solid type, a type environment gives types for free variables, or more precisely, a type environment is a function to transfer varaibles to Types

Let O be a function from ObjectIdentifiers to Types

The sentence O |- e: T

is read: Under the assumption that variables have the types given by O, it is provable that the expression e has the type T

Type environment is passed down the AST from the root towards the leaves, while types are computed up the AST from the leaves towards the root, notice the diffrerece here.
Subtyping

Define a relation called subtyping ≤ on classes
- X ≤ X
- X ≤ Y if X inherits from Y
- X ≤ Z if X ≤ Y and Y ≤ Z
lub(X,Y), the least upper bound of X and Y, is Z if
- X ≤ Z ^ Y ≤ Z, then Z is an upper bound
- X ≤ Z’ ^ Y ≤ Z’ => Z ≤ Z’, then Z is least among upper bounds
the least upper bound of two types is their least common ancestor in the inheritance tree
Typing Methods

M for method signatures, M(C,f) = (T1,…Tn,Tn+1) means in class C there is a method f that f(x1:T1,…,xn:Tn): Tn+1 this is mostly used in dispatch and barely used anywhere else,but it must be added to all rules.

For some cases involving SELF_TYPE, we need to know the class in which an expression appears So we also defined the current class C

which sum up the rule as O,M,C |- e: T An expression e occurring in the body of C has static type T given a variable type environment O and method signatures M
Relation of dynamic type and static type.

Soundness theorem for the Cool type system: for every random E. dynamic_type(E) ≤ static_type(E) which means all operations that can be used on an object of type C can also be used on an object of type C’≤ C
SELF_TYPE

when invoking a method, it allows the method to return type of the class(SELF_TYPE) which invoking it instead of the class contains this method. But remember the declaration of method should change to

method() : SELF_TYPE { … }
- It is a static type
- It helps the type checker to keep better track of types
- It enables the type checker to accept more correct programs
Let T and T’ be any types but SELF_TYPE, and SELF_TYPEc ≤ C
1. SELF_TYPEc ≤ SELF_TYPEc
2. SELF_TYPEc ≤ T if C ≤ T
3. T ≤ SELF_TYPEc always false
4. T ≤ T’
5. lub(SELF_TYPEc, SELF_TYPEc) = SELF_TYPEc
6. lub(SELF_TYPEc, T) = lub(C, T)
7. lub(T, SELF_TYPEc) = lub(C, T)
8. lub(T, T’) = T
SELF_TYPE is not always allowed in Cool
1. class T inherits T’ {…} where T, T’ cannot be SELF_TYPE
2. m@T(E1,…,En) where T cannot be SELF_TYPE
3. m(x : T) : T’ { … } where only T’ can be SELF_TYPE!
Error Recovery

Introduce a new type No_type for use with ill-typed expressions
- Define No_type ≤ C for all types C
- Every operation is defined for No_type (With a No_type result)

Code Generation

Runtime Organization

Assume Organization in memory like a big rectangular bookshelf.
- Layers (from head(low address) to toe(high address))
  - First Layer contains code, for most languages, fixed size and read only
  - Second Layer contains static data with fixed addresses (e.g., global data)
  - Third Layer (Stack) contains activation records for each currently active procedure
  - Bottom Layer (Heap) contains dynamic data which grows to head
Activation

An invocation of procedure P is an activation of P
- The lifetime of an activation of P is
  - All the steps to execute P
  - Including all the steps in procedures P calls
- Notice:
  - Lifetimes of procedure activations are properly nested
  - Activation lifetimes can be depicted as a tree, which is called an activation tree
  - Since activations are properly nested, a stack can track currently active procedures
Activation Records
- The information needed to manage one procedure activation is called an activation record (AR) or frame The compiler must determine, at compile-time, the layout of activation records and generate code that correctly accesses locations in the activation record. Thus, the AR layout and the code generator must be designed together!
- Activation Records usually cantains (e.g: If procedure F calls procedure G)
  - Space for G’s return value
  - F’s actural parameters
  - Pointer to the previous activation record (The control link; points to AR of caller of G)
  - Machine status prior to calling G (Contents of registers & program counters and Local variables)
  - Other temporary values
Code Generation

We simulate stack machine instructions using MIPS instructions and registers
- Stach Machine: a simple model for code generation, for an instruction r = F(a1…an) it
  - Pops n operands from the stack
  - Computes the operation F using the operands
  - Pushes the result r on the stack
- MIPS (originally an acronym for Microprocessor without Interlocked Pipeline Stages) is a reduced instruction set computer (RISC) instruction set (ISA) developed by MIPS Technologies (formerly MIPS Computer Systems, Inc.).
  - Acturally we just need to get familiar with it’s instructions and registers, konwing the def. is useless nor helpful here ;)
- MIPS registers used here are:
  - $a0 accumulator (Q: why call it a0? A: just convention. Notice it’s always on the top of stack)
  - $sp stack pointer, it contains address of the next empty location on the stack
  - $t1 temperary register
  - $fp frame pointer, a pointer to the current activation
  - $ra return address
- MIPS instructions used here are:
  - lw reg1 offset(reg2) –> load 32-bit word from address reg2 + offset into reg1, notice offset is always an integer
  - add reg1 reg2 reg3 –> (reg1 <- reg2 + reg3)
  - sw reg1 offset(reg2) –> store 32-bit word in reg1 at address reg2 + offset
  - addiu reg1 reg2 imm –> reg1 <- reg2 + imm, here imm means immediate value
  - li reg1 imm –> reg1 <- imm
  - sub reg1 reg2 reg3 –> reg1 <- reg2 - reg3
  - beq reg1 reg2 label –> go to branch label if reg1 == reg2
  - b label –> unconditional jump to label
  - jar label –> jump to label, save address of next instruction in $ra
  - jr –> jump and link, jump to address in register reg what means jump and link?
Temporaries

Temporary is a fixed space in a stack for temporary values. Assign a location in the AR for each temporary can make the code much efficient.
- Let NT(e) = # of temps needed to evaluate e:
  - NT(e1 + e2) = max(NT(e1), 1 + NT(e2))
  - NT(e1 - e2) = max(NT(e1), 1 + NT(e2))
  - NT(if e1 = e2 then e3 else e4) = max(NT(e1),1 + NT(e2), NT(e3), NT(e4))
  - NT(id(e1,…,en) = max(NT(e1),…,NT(en))
  - NT(int) = 0
  - NT(id) = 0
- For a AR that has temps, a function definition f(x1,…,xn) = e has 2 + n + NT(e) elements
  - Return address (1)
  - (old) Frame pointer (1)
  - n arguments (n)
  - NT(e) locations for intermediate results (NT(e))
Object Layout

A Cool Object contains: + Class Tag: identifies class of a object + Object Size: size of an object in words + Dispatch ptr: a pointer to the table of pointers points to methods of this object’s class

The offset for an attributes is the same in a class and all of its subclass

The method and args in a class and its subclass is always in a fixed position, even is overwritten, so in execution the args might be different in content, but they are of the same name

Optimization

Optimizing compilers repeat optimizations until no improvement is possible The optimizer can also be stopped at any point to limit compilation time Goal: Maximum benefit for minimum cost

Intermediate Language (IL)
- Intermediate code gather up tp form intermidate language, which is a language between the source and the target, the biggest differece between it and assembly code is it can use any number of registers, but in other aspects it’s almost the same as assembly code thus it is also called a high level assembly code The form of IL is: t1 := y * z. It should be used in optimization phase because:
  - it is machine independent, means it doesn’t need to consider the structure of computer the code is running on.
  - Exposes optimization opportunities clearly
- Basic block: A maximal sequence of instructions written in intermediate language with no labels (except at the first instruction), and no jumps (except in the last instruction)
- Control Flow Graph: use basic blocks as nodes and jumps as vertexs with directions
Optimization types:

There are three granularities of optimizations:
- Local optimizations (Apply to a basic block in isolation, mostly use)
- Global optimizations (Apply to a control-flow graph (method body) in isolation, many use)
- Inter-procedural optimizations (Apply across method boundaries, few use)
Local Optimization

On intermediate Language: It is the simplest form of optimization, it just need to optimize one basic block instead of the whole procedure body, it has these methods:
- Algebraic Simplification:
```
// delete useless code:
x := x + 0
x := x * 1
// simplify statements:
x := x * 0  => x := 0
y := y ** 2 => y := y * y
x := x * 8  => x := x << 3
```
- Constant Folding: (can be dangerous if applied in different machines)
```
x := 2 + 2 => x := 4
if 2 < 0 jump L      // can be deleted
```
- Constant propagation:
```
x := 3          x := 3
y := Z * w  =>  y := Z * w
q := x + y      q := 3 + y
```
- Deadcode Elimination: eliminate unreachable basic blocks
  
  E.g., basic blocks that are not the target of any jump or “fall through” from a conditional
- Single Assignment: each register occurs only once on the left-hand side of an assignment
```
// b is a fresh register
x := z + y      b := z + y
a := x      =>  a := b
x := 2 * x      x := 2 * b
```
- Common Subexpression Elimination:
```
// the values of x, y, and z do not change
// in the … code
x := y + z      x := y + z
…           =>  …
w := y + z      w := x
```
- Copy Propagation:
```
// a should not appear anywhere else in the program
x := z + y      b := z + y      b := z + y
a := x      =>  a := b      =>  x := 2 * b
x := 2 * a      x := 2 * b
```
On Assembly Code: use Peephole Optimization, replace improved version of instructions with original instructions, also need to be applied repeatedly for maximum effect.

Learning Compiler -- Intro, Lexical Analysis and Parsing

Tue, 01 Apr 2014 13:09:00 -0400

This is the summary of Stanford Compiler Lessons on Coursera

Intro

The basic steps of compile is

lexical analysis
parsing
semantic analysis
optimization
code generation

Lexical Analysis

the implementation of lexical analysis contains two things:

first, partition the input string into lexemes

Second, identify the token class of each lexeme

Notice that this method mostly use Left-to-right scan, and lookahead sometimes is required.

Token Class(In COOL)
- whitespace: : a non-empty sequence of blanks, newlines, and tabs
- keyword: “if” or “else” or “then” or …
- identifier: strings of letters or digits, starting with a letter
- integer: a non-empty string of digits
Regular Language

The regular expressions over ∑ are the smallest set of expressions including

In regular languages there are rules descirbed in sets like:
- ℇ (Epsilon)
- ‘c’ (Single Charactor)
- A + B (Union)
- AB (Concatation)
- A* (Iteration)
Formal Language

Let ∑ be a set of characters (an alphabet). A language over ∑ is a set of strings of characters drawn from ∑ It is represeted as L(e) = M, where e stands for regular expression, M stands for set of strings. In formal languages I’ve learned that multiple syntaxs of regular language can only point to one semantic meaning, which is the formal language.
Lexical Specification
- Procedure:
  1. Write a rexp for the lexemes of each token class
  2. Construct formal language L(R), matching all lexemes for all tokens
  3. Let input be x1…xn For 1 ≤ i ≤ n check x1…xn ∊ L(R)
  4. If success, then we know that x1…xi ∊ L(Rj) for some j
  5. Remove x1…xi from input and go to (3)
  6. return ERROR token if all strings are not in the lexical specification
- Notice:
  - one regular expression should match as long as a string can be.
  - regular expression’s priority is determined by the specified order.
  - An Error match should always have the lowest priority
Finite Automata
- A finite automaton consists of
  - An input alphabet ∑
  - A set of states S
  - A start state n
  - A set of accepting states F ⊆ S
  - A set of transitions s1 (on input a) -> s2
- Deterministic Finite Automata (DFA)
  - One transition per input per state
  - No ℇ-moves
  - Faster to execute
- Nondeterministic Finite Automata (NFA)
  - Can have multiple transitions for one input in a given state (which may ends in different results)
  - Can have ℇ-moves
  - In general, smaller than DFA
Implementing Finite Automata
- Procedure: Lexical Specification -> Regular Expressions -> NFA -> DFA -> Table-driven Implementation of DFA
- NFA -> DFA
  - Epsilon Closure: Pick one state, then look at all the states it can reach by following epsilon moves.
  - So basically the transition is about finding epsilon closures in NFA of different inputs, then gather each epsilon closure up as one DFA state.
- A DFA can be implemented by a 2D table T (Acturally NFA can also do the trick,but the table would be huge…)
  - One dimension is states
  - Other dimension is input symbol
  - For every transition Si (on input a) -> Sk, define T[i,a] = k

Parsing

Finite Automata can really only express things where you can count modulus on k, where k is the number of states.

So we need parsing, which transfer sequence of tokens from lexer to parse tree of the program.