Dec 5, 2014

Misc

1. imagemagick is wonderful! Let's say we have img1.png img2.png. This command compares two images and highlight differences into diff.png:


compare -highlight-color 'rgba(255, 255, 0, 0.5)' -lowlight-color 'rgba(255, 255, 255, 0.3)' img1.png img2.png diff.png
This give number of different pixels between two:
convert img1.png img2.png -compose Difference -composite -format '%[fx:mean*100]' info:
2. terminal-notifier Get alert by using command line:
/Applications/terminal-notifier.app/Contents/MacOS/terminal-notifier -message 'Hello' -title 'CI'

3. Strider

A lightweight CI server, using NodeJS, run locally.
Using terminal-notifier, we can use this to run tests and other background tasks.

4. restview

Preview while editing ReST file. Very connivence!

Install:
pip install restview
Example to customise style:
https://gist.github.com/manhg/1696afbc03282466636b

Oct 23, 2014

OS X Yosemite: How to change Japanese default font



In my opinion the default Japanese font in Yosemite is ugly!

Let's replace it with better one.

Fast way: open Terminal and run:
sudo wget http://goo.gl/8APaAA -O /System/Library/Frameworks/ApplicationServices.framework/Frameworks/CoreText.framework/Resources/DefaultFontFallbacks.plist

You need to enter your password.
Restart your computer and see change.

Here is more advanced changes (more customization)
Locate
/System/Library/Frameworks/ApplicationServices.framework/Frameworks/CoreText.framework/Resources/
and make a copy of DefaultFontFallbacks.plist as a backup.
Then copy it your documents to make change using Xcode (we cannot edit it directly)

Inside that, find an entry with "ja".
For sans-serif, HiraKakuPro-W3 is good one.
For serif, HiraMinProN-W3 is nice.

Save and copy DefaultFontFallbacks.plist back to
sudo cp DefaultFontFallbacks.plist /System/Library/Frameworks/ApplicationServices.framework/Frameworks/CoreText.framework/Resources/

We done!
I made a tuned DefaultFontFallbacks.plist for download here:
http://goo.gl/8APaAA


Update:
There is an easier way to do:
https://github.com/schreiberstein/lucidagrandeyosemite

Oct 1, 2014

New tools to my developement

Live Reload X
http://nitoyon.github.io/livereloadx/
This is based on Livereload but having command line interface.
It does auto fetch images/js/css (without reloading)
or auto-reloading (while editing script such as PHP, HTML),
great for doing front-end development.

ZSH
https://github.com/robbyrussell/oh-my-zsh
Alternative to Bash shell. See how awesome it's here:
http://www.slideshare.net/jaguardesignstudio/why-zsh-is-cooler-than-your-shell-16194692

Day-O
http://shauninman.com/archive/2011/10/20/day_o_mac_menu_bar_clock
Although not related to development, this does provides quick calendar, very useful.
Another quick way is using command line, run: $ cal

Arch Linux
http://www.themukt.com/2014/06/07/arch-linux-tutorial/
Always updated, friendly, lightweight and designed for geek.
A great distribution.
Arch's wiki is very well-written.
If only if I knew Arch besides Deiban/Ubuntu, the Linux world will be more open to me.



Sep 27, 2014

Weekend courses on UX / UI

Why UX/UI matters?
https://www.youtube.com/watch?v=O94kYyzqvTc

http://www.goodui.org
GoodUI did a good job on pointing out what kind of design and why they're cool.
It has fundamental tips on designing usable UX/UI.

Sep 17, 2014

Debug Laravel View

Laravel is a very nice framework, except some annoying.

One of it is the famous "Method Illuminate\View\View::__toString() must not throw an exception".
We often found this in the end of an action in a controller:

return View::make('path/view');
This implicitly converts the View to a string, by calling magic __toString() method. If something wrong with the view, we are shielded from seeing what happen.

To get around, use the explicit version:
return View::make('path/view')->render();
You will get your desire and "real" exception caused the view die!

Sep 13, 2014

Compile preprocessors like Coffeescript / SASS on saving in Komodo Edit

Be able to customise the editor is very convenient.

A normal approach is running a "watcher".
This approach is to call the compiler on saving (check Trigger > After save in Marco properties)
The code can be extends for other usage! : )

Note that Komodo has built-in syntax checking for SASS. I want this be off because it generates ".sass-cache" without giving me result. Check it off in preference window.

If something wrong, the command line output will tell you. That's enough : )
We can further "parse" the output to quick click to show the wrong code.

// Run after file saved
// 
// Install Sass: sudo gem install sass
// Install CoffeeScript: npm install -g coffee-script
komodo.assertMacroVersion(3);
var koDoc = ko.views.manager.currentView.koDoc;
var input = koDoc.file.dirName + '/' + koDoc.file.baseName;
var barename = input.substr(0, input.length - koDoc.file.ext.length);

switch (koDoc.file.ext) {
    case '.sass':
        var output =  barename + '.css';
        ko.run.runEncodedCommand(window, 'sass --no-cache ' + input + ' ' + output);
        break;
    case '.coffee':
        ko.run.runEncodedCommand(window, 'coffee --map --compile ' + input);
        break;
    default:
        break;
}

Sep 10, 2014

Japanese form - confirm - complete flow in Perl

I use the same view as both form and validation.
The trick here is to use multiple submit buttons to determine whether the form is confirmed or not.

The sample is written in Perl with module HTML::Template, however, the idea is open for all server side languages (Java, PHP, Python, ...)

Processing Perl script:
my $query = CGI->new;

    # Flag to display form inputs or not
    my $input = 1;
    if ($ENV{'REQUEST_METHOD'} eq 'POST') {
        if (validate_form()) {
            if ($query->param('submit_confirm') || $query->param('submit_confirm.x')) {
                # Confirmed, end here
                send_email();
                print $query->redirect('complete.pl');
                exit 0;
            }
            if ($query->param('submit_edit') || $query->param('submit_edit.x')) {
                $input = 1;
            } else {
                $input = 0;
            }
        } else {
            # Go back, display form to correct
            $input = 1;
        }
    }
    if ($input) {
        $query->param('step_input', 1);
    }
HTML views/form.html (Supposed to use HTML::Template)
Note that submit buttons should have two separate names. Some browsers like Firefox need this to distinguish. We can determine which button chosen by checking if field submit_confirm.x is existed.
It's tricky. Browser sends along a coordinate...
{% input family %}

         <tmpl_if name="step_input">
            <input type="image" src="confirm.gif" alt="確認">
        <tmpl_else>
            <input name="submit_edit" value="edit" type="image" src="edit.gif" alt="戻る">
            <input name="submit_confirm" value="confirmed" type="image" src="complete.gif" alt="登録する">
        </tmpl_if>
A little trick to writing view for both, by customised filter for HTML::Template
Having this, new shiny sugar syntax {% input family %} will render the input box, as well as validation message if found $message_family
sub custom_filter {
    my $text_ref = shift;
    
    # Define a new syntax {% input family %}
    $$text_ref =~ s/\{%\s*input\s+(.*?)\s*\%}
        /<tmpl_if name="step_input">
            <tmpl_if name="message_$1">
                <span class="message">
                    <tmpl_var escape=html name=message_$1>
                <\/span>
            <\/tmpl_if>
            <input 
        <tmpl_else>
            <tmpl_var escape=html name=$1>
            <input type="hidden"
        <\/tmpl_if>
        name="$1"
        value="<tmpl_var escape=html name=$1>">
        /gx;
}
Then register it:
    my $template = HTML::Template->new(
        filename => 'views/form.html',
        associate => $query,
        filter => \&custom_filter,
    );

Sep 1, 2014

Cẩm nang đi tàu ở Nhật

Mục lục:

1. Tàu và tàu điện
只見線 Tuyến tàu với phong cảnh hùng vĩ của miền Tohoku
  (Nguồn: marumine.co.jp)
2. Ga tàu
3. Cấu trúc ga tàu
4. Cửa soát vé
5. Tìm thông tin, tờ rơi, bảng điện tử
6. Tuyến tàu
7. Giờ giấc và sự cố
8. Đi tàu
9. Tra cứu
10. Vé thường, vé tháng, vé đặc biệt
11. Sử dụng máy bán vé
12. Leo lên tàu
13. Làm thế nào biết mình đi đúng hướng?
14. Chuyển tiếp tàu
15. Văn hoá trên tàu
16. Quên đồ trên tàu
17. Tản mạn: tàu điện ngầm, shinkansen.

Chọn nơi sống ở Nhật

Đến một nơi xa lạ, chọn được nơi sống ưng ý đôi khi là may rủi hơn là lựa chọn. Tuy nhiên nếu tìm hiểu kỹ trước khi quyết định sẽ giúp tránh phải chuyển nhà (khá đắt đỏ), chịu đựng (với không gian chưa ưng ý).

Aug 24, 2014

More on Komodo Edit macros

Using git grep inside 

It's a very great experience.
To set up, fill following as in images:

^(?P.+?):(?P\d+):(?P.*)$
git grep %(w:orask:Search Term) -- "%(ask:File Pattern:*)"
The result will be parsed and display in Komodo-style find results. I can even click on it to open the files!
Right click project structure to to mkdir is not a straight way at all. Finally I was able to create a folder using keyboard by this JS macro! I set it to Shift + Cmd + N
var koView =  ko.views.manager.currentView;
if (koView) {
    var folder = koView.koDoc.file.dirName;
    var names = ko.dialogs.prompt("New folder at: " + folder);
    if (!names) return;
    var path = folder + '/' + names;
    ko.run.runEncodedCommand(window, 'mkdir -p "' + path + '"');
} else {
    ko.places.viewMgr.addNewFolder();
}
// Another way - reference: https://github.com/Komodo/KomodoEdit/blob/trunk/src/components/koIOs.idl
// var os = Components.classes["@activestate.com/koOs;1"].getService(Components.interfaces.koIOs); 
// os.mkdir(path);
By the similar way, I also created 1 macro for creating file within current project folder or current file folder (if available)
var koView =  ko.views.manager.currentView;
if (koView) {
    var folder = koView.koDoc.file.dirName;
    // Reference: https://github.com/Komodo/KomodoEdit/blob/trunk/src/views/koIFileEx.idl
    var file = Components.classes["@activestate.com/koFileEx;1"].createInstance(Components.interfaces.koIFileEx);
    var name = ko.dialogs.prompt("New file at: " + folder);
    if (!name) {
        return;
    }
    var path = folder + '/' + name;
    file.URI = ko.uriparse.pathToURI(path);
    file.open("w");
    file.close();
    ko.open.URI(path);
} else {
    ko.places.viewMgr.addNewFile();
}

Jul 8, 2014

Relative date format in Japanese

/*
 * Return relative date format in Japanese, eg: 5時前
 * max param indicates when to return absolute datetime
 * Default is 30 days
 */
function ja_relative_time($stamp, $max = 2592000) {
    $interval = date_create('now')->diff(new DateTime($stamp));
    if ($interval->s >= $max) {
        return $stamp;
    }
    $suffix = ( $interval->invert ? '前' : '' );
    if ( $v = $interval->y >= 1) return $interval->y . '年'. $suffix;
    if ( $v = $interval->m >= 1) return $interval->m . '月'. $suffix;
    if ( $v = $interval->d >= 1) return $interval->d . '日'. $suffix;
    if ( $v = $interval->h >= 1) return $interval->h . '時'. $suffix;
    if ( $v = $interval->i >= 1) return $interval->i . '分'. $suffix;
    return $interval->s . '秒'. $suffix;
}

Jun 27, 2014

Nginx proxy cache setting for Moin

MoinMoin is a nice wiki. I tried to cache it, and luckily succeed:

# Save this, e.g.: /etc/nginx/proxy_cache for reusable
proxy_cache zone-cache;

# Should add more if
proxy_cache_key "$host$request_uri";
proxy_cache_valid any 1d;
proxy_cache_methods GET HEAD;
proxy_ignore_headers Set-Cookie Cache-Control Expires;
proxy_hide_header Cache-Control;
proxy_hide_header Expires;
proxy_http_version 1.1;

server {
     proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=zone-cache:4m max_size=1000m;    
     location @moin {
        include proxy_cache;
        proxy_pass http://127.0.0.1:8080;
        # each application need specify when to by pass cache
        set $no_cache 0;
        if ($query_string ~* "action=") {
            set $no_cache 1;
        }
        if ($http_cookie ~* "MOIN_SESSION_") {
            set $no_cache 1;
        }
        proxy_cache_bypass $no_cache;
    }
    location / {
        try_files $uri @moin;
    }
    location /moin_static197/ {
        alias /srv/moin/web/static/htdocs/;
    }
    # more here
}

Jun 23, 2014

Komodo Tips

Komodo Edit, an code editor based on Mozilla, has a nice features called "Macro", allows programming of UI. This applies to version 8.5 in MacOSX.
First, open Toolbox sidebar, right click / click on Setting button in top-right corner, we will have a list of customisation available (command, macro, snippet).

My favourite list:

1. View current file in Shift JIS

Japanese legacy code are largely written in Shift JIS encoding.
Adding this Javascript macro let you quickly switch:
komodo.assertMacroVersion(3);
var koDoc = ko.views.manager.currentView.koDoc;
if (koDoc.encoding.short_encoding_name != "Shift JIS") {
    koDoc.setEncodingFromEncodingName("Shift JIS");
}
I often assign Cmd + Shift + J shortcut to this.

2. Open GitX for this file

Git should be init beforehand (or you can add a command for init : )
The "%D" represents current folder of current file.
Add a command look like below, and set your own shortcut key:

Based on this, you can easily create a shortcut for executing PHP, Python script:

3. Move the current line up / down

The days before I works with Notepad++, above useful shortcuts are missing in Komodo.
However, we can "emulate" it using 2 macros.
Script below are taken from Komodo forums:
http://community.activestate.com/forum/move-line-move-line-down-macros

For move selection / current line down
// Move Line or Move Selection Down
komodo.assertMacroVersion(3);
if (komodo.view) { komodo.view.setFocus() };
var ke = komodo.editor;

if( ke.lineFromPosition( ke.currentPos ) == ( ke.lineCount - 1 ) )
        return;

// Prevent Undo remember macro steps
ke.beginUndoAction();
   
var sel_start_a = ke.selectionStart;
var sel_end_a = ke.selectionEnd;

// Extend selection to beg of line at front and end of line at back
var selStartLine = ke.lineFromPosition(ke.selectionStart);
var selEndLine = ke.lineFromPosition(ke.selectionEnd);
var numLinesSelected = selEndLine - selStartLine;

var selStart = ke.positionFromLine( selStartLine );
var selEnd   = ke.getLineEndPosition( selEndLine );

ke.setSel( selStart, selEnd );

// Determine original selection position offset related to extended
var sel_start_b = ke.selectionStart;
var sel_end_b = ke.selectionEnd;
var offset_start = sel_start_a - sel_start_b;
var offset_end = sel_end_b - sel_end_a;  

// Copy the selected text and remove it
var text =  komodo.interpolate('%s');
komodo.doCommand('cmd_delete'); // This leaves a blank line in place of selection

// Move our selection to a new place
// First move our blank line up
komodo.doCommand('cmd_lineNext')
ke.lineTranspose();

// Insert our text
ke.insertText(ke.currentPos, text);

// Restore selection            
var newSelStartLine = ke.lineFromPosition( ke.currentPos );
var newSelEndLine   = newSelStartLine + numLinesSelected;

var newSelStart = ke.currentPos + offset_start;
var newSelEnd   = ke.getLineEndPosition(newSelEndLine) - offset_end;

ke.setSel(newSelStart, newSelEnd);

// End of prevent Undo
ke.endUndoAction();

For moving up:

// Move Line or Move Selection Up
komodo.assertMacroVersion(3);
if (komodo.view) { komodo.view.setFocus() };
var ke = komodo.editor;

if( ke.lineFromPosition( ke.currentPos ) == 0 )
        return;

// Prevent Undo remember macro steps
ke.beginUndoAction();
   
var sel_start_a = ke.selectionStart;
var sel_end_a = ke.selectionEnd;

// Extend selection to beg of line at front and end of line at back
var selStartLine = ke.lineFromPosition(ke.selectionStart);
var selEndLine = ke.lineFromPosition(ke.selectionEnd);
var numLinesSelected = selEndLine - selStartLine;

var selStart = ke.positionFromLine(selStartLine);
var selEnd   = ke.getLineEndPosition(selEndLine);

ke.setSel(selStart, selEnd);

// Determine original selection position offset related to extended
var sel_start_b = ke.selectionStart;
var sel_end_b = ke.selectionEnd;
var offset_start = sel_start_a - sel_start_b;
var offset_end = sel_end_b - sel_end_a;  

// Copy the selected text and remove it
var text =  komodo.interpolate('%s');
komodo.doCommand('cmd_delete'); // This leaves a blank line in place of selection

// Move our selection to a new place
// First move our blank line up
ke.lineTranspose();
komodo.doCommand('cmd_linePrevious')

// Insert our text
ke.insertText(ke.currentPos, text);

// Restore selection
var newSelStartLine = ke.lineFromPosition( ke.currentPos );
var newSelEndLine   = newSelStartLine + numLinesSelected;

var newSelStart = ke.currentPos + offset_start;
var newSelEnd   = ke.getLineEndPosition(newSelEndLine) - offset_end;

ke.setSel(newSelStart, newSelEnd);

// End of prevent Undo
ke.endUndoAction();

4. Find and replace

The find window is often get lost because of Windows overlap.
A solution is clicking pin icon on the right down corner.
Also, set proper include and exclude pattern may help while searching on a directory.
My exclude pattern is:
vendor:.svn:.git:template_c:*~
The regex in replace box using notation \1, ..  \n for matched groups.

5. Emmet

Emmet is an add-on for typing HTML faster.
Watch how amazing it is at http://docs.emmet.io/
I often set key Shift - Enter for expanding emmet abbr.

6. Keybindings
Key with * are not set by default.

Cmd + Shift + K = Invoke tool
*Cmd + P = Quick open (like Sublime does, but we need to create a "project" beforehand)
*Cmd + | =  Browser preview split view: the page reload on save
*Cmd + M = Duplicate line or selection
*Cmd + G = Go to line
*F4 and Esc = Complete word
*Cmd + K = Set mark
*F10 = Show current file in places




Jun 20, 2014

MacOSX good apps list

Applications

* Komodo Edit: Main editor

* Textmate2: Additional Editor

* Homebrew: Build packages and software for OSX

* iterm2: Terminal, quickly copy / paste; window-division.

* SequelPro: For managing MySQL (local and remote). For Postgres, SQLite: adminer (PHP based) is a good choice.

http://ohmyz.sh/ improve shell

Visual Git, line by line staging.
For diff, using console-based colordiff is good.
Using sdiff for side-by-side diff

* Adium: Messenger for Facebook, Google, Yahoo combined. Messages/Facetime from Apple are good as well.

* FileZilla: Solid SFTP/FTP client or Cyberduck: many features

* SketchBook Express: freely and powerful drawing tool

* Seashore: bitmap image tool

* MacGDBp: Standalone PHP Deugger GUI

* VirtualBox: VM and testing

* LibreOffice

Utilities:

* Caffeine: Prevent sleep

* PCKeyboarHack, Keyremap4Macbook: change keymap

* Spectacle: Windows arrangement tool (change size/position faster)

* Clipmenu: clipboard history and snippet tools (great time saving)

* Google Japanese Input: better IME

* LICEcap: GIF screenshot win animation

* Developer color picker http://download.panic.com/picker/

http://sequentialx.com View bulk images

* ToyViewer: fast images viewer

* imageoptim Reduce images size (replaces)

* nvALT: notetaker http://brettterpstra.com/projects/nvalt/

Jun 16, 2014

MySQL script to generate a ASCII-table of a database schema

Run this in the command line client (interactive mysql program):
-- References: http://dev.mysql.com/doc/refman/5.0/en/columns-table.html
use information_schema;
select table_name 'Table', column_name 'Column', column_key 'Index', column_type 'Type',
if (IS_NULLABLE = 'NO', if(column_default IS NULL, '', column_default), 'NULL') 'Default', column_comment 'Comment'
from columns where table_schema = '$DB_NAME';
Replace $DB_NAME by your own name. Japanese version:
-- References: http://dev.mysql.com/doc/refman/5.0/en/columns-table.html
use information_schema;
select table_name 'フィール', column_name '論理名', column_key 'インデックス', column_type '型',
if (IS_NULLABLE = 'NO', if(column_default IS NULL, '', column_default), 'NULL') 'デフォルト', column_comment '備考'
from columns where table_schema = '$DB_NAME';

Jun 15, 2014

VPS Providers in Japan

Most providers requires credit cards, however there are some with flexible bank transfers as payment method:

http://www.cloudcore.jp
Backed by KDDI

Other choices:
http://dream.jp/vps/
http://www.kagoya.jp/cloud/vps/

Most plans start at 500 yen/month, fairly cheep with good config.

May 28, 2014

Interesting points of Python

After 4 months learning and its ecosystem I felt the following amazing things in Python:
(created or borrowed from other language)

1. List comprehension
Quick and beauty way to work with array of elements:
>>> vec = [2, 4, 6]
>>> [3*x for x in vec]
[6, 12, 18]
More: https://docs.python.org/3.4/tutorial/datastructures.html#list-comprehensions

2. Decorator
A function, which wraps another function, is called "decorator", enable us to write:
@app.route('/')
def hello_world():
    return 'Hello World!'

More: http://www.jeffknupp.com/blog/2013/11/29/improve-your-python-decorators-explained

3. Iterator & generator
Good for memory allocation and nice syntax.
https://speakerdeck.com/pyconslides/transforming-code-into-beautiful-idiomatic-python-by-raymond-hettinger-1

4. List unpacking and indexing

5. Named-arguments in functions
It's make the code more readable, without length like Objective-C

6. Collections
defaultdict, namedtuple, ... are our friends!

7. Context manager
It's "with" structure.

More reference:
http://stackoverflow.com/questions/101268/hidden-features-of-python
http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html
http://www.cs.cmu.edu/~nschneid/pythonathan/

May 16, 2014

Push an existing git repos to a SVN repos

The old thing (SVN) and new thing (Git) sometimes are forced to co-exist.

Assume we are at local repos work folder, here are the trick (rebase git-svn):

$ git svn init http://svn/path
$ git svn fetch
$ git rebase git-svn
$ git svn dcommit
From now on, to push to SVN, issue:
git svn dcommit 
to pull from SVN:
git svn rebase

May 13, 2014

SQLite

I got "unable to open database file" when trying to make changes in an SQLite database using PHP PDO.

Finally the real problem is:

If we have /path/name.db writable, it's NOT enough!
We have to make the container dir of database file (this case, /path/ ) be writable AS WELL.

Tricky! I guess SQLite need temporary space in this dir.

Apr 17, 2014

JS OOP with publish/subscribe custom events using jQuery

As the Javascript script grows, traditional programming style will result in messy code. With the help of custom event and OOP, we can keep the code clean, easy to extends and maintain. Below is the snippet comes from my lesson:
<script type="text/javascript" src="http://code.jquery.com/jquery-1.11.0.min.js"></script>
<script type="text/javascript">
// For testing
function puts(text) {
    document.write("<br>" + text);
}

// Thanks to http://javascript.info/tutorial/all-one-constructor-pattern
function BaseClass() {
 // Private, invisible to child class and outside
 var x;

 // In lost context, using "self" to reference
 var self = this;

 // Public
 this.a = 'a freely property';

 // Public access to restricted var
 this.propertyX = function(_x) {
  if (typeof _x != 'undefined')  x = _x;
  return x;
 }
 this.actionY = function(message) {
  // Bla bla
  // "Publish" an event, pass params as an array
  $(self).triggerHandler('custom_event', [message]);
 }
 $(self).on('custom_event', function(event, message) {
  puts("Base: " + message);
 });
}

function ChildClass() {
 // Inherit, able to have many parents
 BaseClass.apply(this);
 var self = this;
 // To call parent implementation, keep a reference to it
 var parent = { actionY: this.actionY }

 this.actionY = function(message) {
  // Super call, note the params as an array
  parent.actionY.apply(this, [message]);
  // Customize here ...
 }

 this.actionZ = function() { /* Other.. */ }

 $(self).on('custom_event', function(event, message) {
  puts("Extended: " + message);
  puts("With access to: " + self.a);
 });
}

// Now it's time to play with class
c = new ChildClass();
c.propertyX(10);
puts(c.propertyX());
c.actionY('JS OOP is tricky');
</script>

Apr 16, 2014

Snippets for backup/deploy Postgres DB

Create dump or schema definition:
pg_dump --no-owner --no-acl -Fc db > db.dump
pg_dump --no-owner --no-acl --schema-only db > db.sql
Restore, deploy:
pg_restore -Fc -C db.dump | psql

Apr 5, 2014

Server-level cache

We often use "plugins" like "WP Super Cache"... to have our contents using less server resources.
We also have to write various PHP-level code to having such functionality.

However, there is an alternative way to do it *automatically*, without having to repeatedly writing code to cache our content at PHP-level.

We care about 4 aspects of caching:
1-Where to cache (often files)
2-How long it cached (says 5 min or 30days, depend on how freshness the website)
3-Bypass for dynamic sections (login, admin, realtime...)
4-Clear cache

Assuming we using Nginx and passing request to backend PHP using socket, typically we having:

proxy_pass http://127.0.0.1:9000

We added something like:
http {
   proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=zone-a:8m max_size=1000m inactive=600m;
}
server {
  proxy_pass http://127.0.0.1:9000
  proxy_cache_key $proxy_host$request_uri$is_args$args;
  proxy_cache_valid 200 302 5m;
  proxy_cache zone-a;
}

proxy_cache_path defines place to put our cache data, naming "zone". We have control over its size, how to creating sub- directories (levels=1:2 )

proxy_cache specifies which zone to store cache.

proxy_cache_key define the pattern to cache. We may need to add how to know that the page is dynamic generated, such as cookie. Or we may use proxy_cache_bypass to tell, example, wp-admin will be bypassed.

set $wp_bypass 0;
if ($request_uri ~ "wp-admin") {
    set $wp_bypass 1;
}
proxy_cache_bypass $wp_bypass;
To clear cache, simply deletes files under zone directory!

Apache also having similar features, and if we need more flexible, a mature cache engine like Varnish may help us. Speed up and concentrate more on development!

Mar 27, 2014

Ngành công nghệ thông tin

Trong ngành Công nghệ thông tin có một số "chức danh" nổi bật - nghề nổi bật sau.

Lập trình viên: là người tiếp cận, hiện thực hóa các ý tưởng bằng kỹ năng lập trình và hiểu biết hệ thống.
Thường hầu hết sẽ là các "anh chàng cận", bạn sẽ ngồi cả ngày với cái máy tính, lõ mọ đọc tài liệu (10%) gõ "code" (20% thời gian), gỡ lỗi (50%), đi uống cà phê (% thời gian còn lại) :)

Kỹ sư: Ở mức độ cao hơn lập trình viên - kỹ sư là người thiết kế. Vạch ra phương hướng, làm các bản đặc tả yêu cầu và kiểm soát chất lượng, kết quả lập trình. Đôi khi người kỹ sư cũng làm một phần công việc của người lập trình ở các giai đoạn khó. Ở mức độ cao hơn, kỹ sư sẽ là trưởng nhóm phát triển, trưởng dự án hay các cấp độ quản lý ở tầm chiến lược hơn.

Kỹ sư cầu nối: tức là kỹ sư có khả năng giao tiếp bằng 2 hay nhiều ngôn ngữ.
Hiện nay mô hình kinh doanh của một số công ty lấy đối tác và nguồn dự án ở nước ngoài. Khi đó công ty sẽ cử người đi làm việc với khách hàng, lắng nghe họ, truyền đạt trở về các thành viên khác không trực tiếp.
Phải là kỹ sư để có thể hiểu được và hỗ trợ tức thì các vấn đề kỹ thuật.

An interesting Nginx config

Limit the number of connections per the defined key. Example: key = IP address:
http {
    limit_conn_zone $binary_remote_addr zone=addr:10m;
}

server {
    location /download/ {
    limit_conn addr 1;
}
http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html

Bắt đầu Linux

Linux là một hệ điều hành có nhiều tính năng hay và phù hợp với học tập, nghiên cứu. Tuy nhiên nó xấu và khá khó dùng.

Để bắt đầu học, cần cài đặt Puppy Linux cùng sử dụng với Windows:
http://puppylinux.org/

Nếu muốn chạy trên máy tính cũ, trên USB, máy ảo (ví dụ VirtualBox), thì phiên bản Slitaz sau khá thích hợp (khoảng 40B). Nó cũng có thể chạy từ USB/CD và cài đặt được.
Tải về:
http://mirror.slitaz.org/iso/4.0/slitaz-4.0.iso

Để cài lên USB, tải chương trình sau và làm theo chỉ dẫn:
http://www.objectif-securite.ch/slitaz/tazusb.exe
Ngoài ra có thể sử dụng các phiên bản Ubuntu khá phổ biến.

Địa chỉ tải gốc:

http://mirror.slitaz.org/iso/4.0/slitaz-4.0.iso

Bắt đầu học Python

Python là ngôn ngữ đơn giản, dễ dùng, dễ đọc!
Python cũng là ngôn ngữ được dùng nhiều nhất ở Google.

Để bắt đầu, trên Windows, tải về:

http://www.python.org/ftp/python/3.3.4/python-3.3.4.msi
Để khám phá và tập lập trình, có thể dùng chương trình sau:
https://code.google.com/p/pyscripter/downloads/detail?name=PyScripter-v2.5.3-Setup.exe&amp;can=2&amp;q=

Với Linux và Mac: Python 2 đã có sẵn.

Hướng dẫn ngắn gọn về học Python
http://en.wikibooks.org/wiki/Non-Program...r_Python_3

Mar 24, 2014

Python DBAPI2

It took me a hour to get rows updated using Python on MySQL, specially PyMySQL driver. The problem is the query is not auto-committed. I have to add a line at the end:
cursor.execute('commit')
Moreover, quoting with HTML data is in trouble, having to do it manually. An important note is, all queries are pre-fetched by default. That mean all data was in RAM no matter we call fetchone or fetch all. In order to save memory, we have to use SSCursor:
cursor = conn.cursor(pymysql.cursors.SSCursor)
Here is a skeleton, querying note field and update it:
connection = pymysql.connect(host='127.0.0.1', user='root', passwd='', db='name', charset='utf8')
cursor = connection.cursor()
cursor.execute('SELECT id, note FROM products')
rows = cursor.fetchall()
for row in rows:
    # new_note = xyz(rows[0])...
    sql = "UPDATE products SET note='%s' WHERE id=%s" % (connection.escape_string(new_note), row[0])
    cursor.execute(sql)
cursor.execute('commit')
cursor.close()
connection.close();

Mar 12, 2014

Stdout UTF8 in Python3

import codecs
import sys
sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())

Mar 6, 2014

In clause in SQL

If we execute the following with expected thousand rows in subquery result, it make takes a while (some minute):
DELETE FROM table 
WHERE row_id IN (
    SELECT ref_id IN ref_table 
    WHERE a_col = '..'
)
I discovered the reason for slowness: IN clause run ineffective with non indexed items So here is a solution:
CREATE TABLE tmp_ref AS
    SELECT ref_id IN ref_table 
    WHERE a_col = '..';

DELETE FROM table 
WHERE row_id IN (SELECT tmp_ref IN tmp_ref);

DROP TABLE tmp_ref;

Run Python as service

Updated: in 2016+, it's the best to use Systemd


Using Python event-driven such as Tornado, we can develop a great system as an alternative to NodeJS technology. However, keep it running as a system service is not built-in. We have to managed ourself to do this. Luckily there is Supervisord http://supervisord.org , written in Python, and greatly get our service in Python up. It can serve not only Python, but almost every program. The installation is trivial in Ubuntu with apt-get install supervisor. Other distro can use pip install supervisor. Below is the case of Ubuntu server installation. We need to create a conf file at /etc/supervisor/conf.d/, as example a-svc.conf (noted the file extension)
[program:a-svc]
process_name=a-svc
user=www-data
dicretory=/srv/svc/
command=/usr/bin/python3 -u -m tornado.autoreload /srv/svc/service.py
# Two setting below is interesting
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/var/log/supervisor/svc.log
Then bring all up running with sudo service supervisor restart Assume our service running at 9876 We can test using curl http://127.0.0.1:9876/a-query To public this service, we can add some kind of config in Nginx:
server {
    # For public access
    location /api/ {
     rewrite ^/api/(.*)$ /$1 break;
     proxy_pass http://127.0.0.1:9876;
 }
}
We have it done nicely!


Mar 4, 2014

Tunnel local web interface

Assuming we have a local web admin (e.g. Solr Admin on 8833), we want to view it via our browser. Because it's not exposed as public server, we will tunnel via SSH:

Using a server target port from localhost local port, then access via http://localhost:{local_port}
ssh -L {local_port}:localhost:{target_port} {user}@{host}

Mar 3, 2014

Auto resize and optimize images

ImageMagick helps us with command utility: mogrify. It replaces original files, so be careful with this.

The trick is how to search for newly upload files. Here is a solution:

# Create a file to compare (default is now)
# A manual options is: touch --date "2014-01-01"
touch /tmp/fcmp~
# Process newly added files
find -type f -newer /tmp/fcmp~ -name *.jpg -exec mogrify -resize 1200x1200 {} \;

Laravel resources

http://laravel-recipes.com/

Mar 2, 2014

For backup: Auto commit Git script

I'm a novice in Bash scripting. But after trying to write this by myself, I found Bash is really useful and interesting as well.
#!/bin/bash
# ------------------------------------------------------
# Git: Auto-commit on scheduled
# ------------------------------------------------------
# Path to list of sites
# Sites should be under git versioning
# Example: /srv/site-a should have:
#
#          /srv/site-a/public <-- HTTP server point here
#          /srv/site-a/.git
SRV_DIR="/srv"
# ------------------------------------------------------
SRV_LIST=$(ls $SRV_DIR)
for s in $SRV_LIST; do
    # Is a dir with git inside?
    if [[ -d $s && -d $s/.git ]]
    then
        echo "Found: $s"
        GIT_BASE=" git --git-dir=$SRV_DIR/$s/.git --work-tree=$SRV_DIR/$s "
        IS_CHANGE=$($GIT_BASE status | grep 'Changes not staged' | wc -l)
        if [ $IS_CHANGE -eq '1' ]
        then
            CURRENT_TIME=$(date)
            $GIT_BASE add $SRV_DIR/$s
            $GIT_BASE commit -m"Auto-commit $CURRENT_TIME"
            $GIT_BASE push backup
        fi
    fi
done

A backup strategy for websites

This article is about an attempt to backup files and MySQL databases on a Linux host.

1. First, create a local git repos, called /git. Set permission: 700 unreadable, except the ower.

2. Each sites, use versioning:
cd /srv/web-a # Your site is located here
git init # This is the real, production file
git init --bare /git/web-a # This is a local backup
git add .
git remote add /git/web-a # Set path
git commit -m"Initial"
git push backup master
In order to secure .git folder, the domain mount point in HTTP server must be like /srv/web-a/public (in a subfolder).Otherwise, we need deny access to .git from webserver.

3. Whenever you make changes, do a commit and push it to backup repos (keep it updated!). We can even schedule to git on daily using cron (done later). This keep uploads and data are all the lastest in repos!

4. Database will be stored in /git/db (althought it's not a repos :D )
We need create an user for dumping, with minimal permission as needed:
CREATE USER 'dump'@'localhost' IDENTIFIED BY 'your-password';
GRANT LOCK TABLES, REFERENCES, SELECT, SHOW VIEW ON *.* TO 'dump'@'localhost';

5. Make a script at /home/you/backup.sh, chmod 700 (runable and readonly for you):
mysqldump --lock-tables=false -u backup -p"your_password" --all-databases \
        | bzip2 > /git/db/all-$(date +%F).bz2
find /git/db/* -mtime +15 -exec rm {} \;
cd /srv/web-a
git add .
git commit -m"Auto-commit at $(date +%F)"
git push backup master

We can customize to dump each of your desire database.
The most important option is --lock-tables=false. It helps websites keep runing while dumping.
The second line is used to remove old backup aged more than 15 days.

6. Finally we need a cron to run:
0 23 * * * /home/you/backup.sh 2>&1 >/dev/null
This will run on everyday at 23:00. Enjoy you night without worry! : )

7. When we are in trouble, we can use powerful git to restore the version we want. Currently database dump is not optimized, because it's a full backup. If there is a solution for incremental backup, it's better to save space and able to archive longer timespan.

Feb 4, 2014

Compile Mathex (A Latex CGI to generate image)

Mathtex from http://www.forkosh.com/mathtex.html

If we careless compile, we can easily produced the error:
cc: error: –DSOMETHING="xyz": No such file or directory
Which really not makes sense at all.
Actually, it maybe some switches is not supported, or the value is not correctly quoted.

I made it worked with Nginx by using fcgiwrap
       location ~ ^/tex.cgi$ {
                gzip off;
                fastcgi_pass unix:/var/run/fcgiwrap.socket;
                include fastcgi_params;
        }
Complie:
cc mathtex.c -o tex.cgi -DDVIPNG=\"/usr/bin/dvipng\" -DLATEX=\"/usr/bin/latex\" -DCACHE=\"cache\/\" DWORK=\"cache\/\"

The most important switches are DCACHE (set up relative cache dir) and DWORK (set up temporary place for the CGI works). They should both writable by the script tex.cgi under fcgiwrap's run user.

Then place stuff using at this structure:
/public
          /cache
          tex.cgi



Jan 29, 2014

Github

If using HTTPS link, we cannot utilize public key auth for Github.
So, using this URL pattern to clone your repository:

ssh://git@github.com/username/reposname.git