海运的博客

PHP通过Yahoo Content Analysis API生成tag

发布时间:June 27, 2014 // 分类:PHP // No Comments

<?php
$text = 'This domain name expired on 12/6/2014 and is pending renewal or deletion.';
$query = "select * from contentanalysis.analyze where text = '".$text."'";
$url = 'http://query.yahooapis.com/v1/public/yql';
$yql_query_url = $url . "?q=" . urlencode($query);
$yql_query_url .= "&format=json";
$yql_query_url .= "&enable_categorizer=true";
$yql_query_url .= "&diagnostics=false";
$yql_query_url .= "&related_entities=true";
$yql_query_url .= "&show_metadata=true";
$ch = curl_init($yql_query_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
$result = curl_exec($ch);
curl_close($ch);
$result = json_decode($result);
var_dump($result);
if(!is_null($result->query->results)) {
  foreach($result->query->results->entities as $word) {
    if (is_array($word)) {
      foreach($word as $subword) {
        echo $subword->text->content."\n";
      }
    } else{
      echo $word->text->content."\n";
    }
  }
}
?>

文档:https://developer.yahoo.com/search/content/V2/contentAnalysis.html

Python使用Selenium/PhantomJS

发布时间:June 26, 2014 // 分类:Python // No Comments

Windows下安装setuptools和pip:
https://bootstrap.pypa.io/ez_setup.py
https://bootstrap.pypa.io/get-pip.py

python ez_setup.py
python get-pip.py

安装selenium:

pip install selenium

安装PhantomJS:

https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.7-linux-x86_64.tar.bz2
tar jxvf phantomjs-1.9.7-linux-x86_64.tar.bz2
cp phantomjs-1.9.7-linux-x86_64/bin/phantomjs /bin/
chmod 755 /bin/phantomjs 

使用示例:

from selenium import webdriver
driver = webdriver.PhantomJS()
driver.get("http://www.baidu.com")
data = driver.title
print data

通过Remote Selenium Server:

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
driver = webdriver.Remote(
  command_executor='http://192.168.1.3:4444/wd/hub',
  desired_capabilities={'browserName': 'PhantomJS',
                                  'version': '2',
                                  'javascriptEnabled': True})
driver = webdriver.Remote(
   command_executor='http://192.168.1.3:4444/wd/hub',
   desired_capabilities=DesiredCapabilities.PHANTOMJS)
driver.get("http://www.baidu.com")
data = driver.title
print data

PhantomJS和Firefox速度对比:

import unittest
from selenium import webdriver
import time
class TestThree(unittest.TestCase):

    def setUp(self):
        self.startTime = time.time()

    def test_url_fire(self):
        self.driver = webdriver.Firefox()
        self.driver.get("http://www.qq.com")
        self.driver.quit()

    def test_url_phantom(self):
        self.driver = webdriver.PhantomJS()
        self.driver.get("http://www.qq.com")
        self.driver.quit()

    def tearDown(self):
        t = time.time() - self.startTime
        print "%s: %.3f" % (self.id(), t)
        self.driver.quit

if __name__ == '__main__':
    suite = unittest.TestLoader().loadTestsFromTestCase(TestThree)
    unittest.TextTestRunner(verbosity=0).run(suite)

PHP使用Selenium

发布时间:June 26, 2014 // 分类:PHP // No Comments

overviewSelenium.png
通过composer安装phpunit和Selenium,先安装composer:

curl -sS https://getcomposer.org/installer | php --install-dir=/usr/bin/

composer.json:

{
  "require": {
    "facebook/webdriver": "dev-master",
    "phpunit/phpunit": "*"
  }
}

安装:

composer.phar install

安装JAVA,安装Selenium Server,启动:

wget http://selenium-release.storage.googleapis.com/2.42/selenium-server-standalone-2.42.2.jar
java -jar selenium-server-standalone-2.42.2.jar

启动浏览器需X环境支持,可使用XVNCX Window
可以使用Firefox扩展Selenium IDE: PHP Formatters录制脚本。
示例脚本:

<?php
class GitHubTest extends PHPUnit_Framework_TestCase {

    protected $webDriver;

    public function setUp()
    {
        $capabilities = array(\WebDriverCapabilityType::BROWSER_NAME => 'firefox');
        $this->webDriver = RemoteWebDriver::create('http://localhost:4444/wd/hub', $capabilities);
    }

    protected $url = 'https://www.baidu.com';

    public function testGitHubHome()
    {
        $this->webDriver->get($this->url);
        var_dump($this->webDriver->getTitle());
    }    
}
?>

不使用phpunit:

<?php
require 'vendor/autoload.php';
$capabilities = array(\WebDriverCapabilityType::BROWSER_NAME => 'firefox');
$webDriver = RemoteWebDriver::create('http://localhost:4444/wd/hub', $capabilities);
$url = 'https://github.com';
$webDriver->get($url);
var_dump($webDriver->getTitle());
?>';

自定义:

require 'vendor/autoload.php';
$profile = new FirefoxProfile();
$profile->setPreference('browser.startup.homepage', 'https://github.com/facebook/php-webdriver/');
$profile->setPreference("general.useragent.override", "Mozilla/5.0");
$profile->addExtension('./vimperator-3.8.2-fx.xpi');
$caps = DesiredCapabilities::firefox(); 
$caps->setCapability(FirefoxDriver::PROFILE, $profile); 
$caps->setCapability(WebDriverCapabilityType::PROXY, array('proxyType' => 'manual', 'httpProxy' => '127.0.0.1:2043'));  
$webDriver = RemoteWebDriver::create('http://localhost:4444/wd/hub', $caps);

文档:
https://github.com/facebook/php-webdriver/wiki
http://facebook.github.io/php-webdriver/namespaces/default.html

Centos下Xvfb配合x11vnc搭建VNC Server

发布时间:June 26, 2014 // 分类:CentOS // No Comments

远程运行Linux窗口程序使用X Windows太重量级了,可以使用Xvfb新建虚拟X窗口,通过x11vnc启动VNC Server并转发Xvfb启动的虚拟窗口。

yum install xorg-x11-server-Xvfb
yum install x11vnc
#新建X虚拟窗口
Xvfb :1 -screen 0 1024x768x24 -nolisten tcp &
#设置默认窗口为新建的虚拟窗口,打开窗口程序时调用
export DISPLAY=:1
#或
DISPLAY=:1 firefox

INIT:

#!/bin/bash
#chkconfig: 345 95 50
#description: Starts xvfb on display 1
if [ -z "$1" ]; then
    echo "`basename $0` {start|stop}"
    exit
fi   
case "$1" in
    start)
    Xvfb :1 -screen 0 1024x768x24 -nolisten tcp &
    export DISPLAY=:1
    echo 'export DISPLAY=:1' >> ~/.bashrc 
    ;; 
    stop)
    killall Xvfb
    ;;
esac

新建VNC服务器并转发指定X窗口

x11vnc -listen 0.0.0.0 -rfbport 5900 -noipv6 -passwd password -display :1 -forever

然后通过VNC客户端连接,默认端口5900,Windows下可使用TightVNC或UltraVNC。
启动的软件窗口太小,设置:

xdotool search --name ".*Mozilla Firefox" windowsize 1440 900

OpenWRT交叉编译安装使用shadowsocks

发布时间:June 25, 2014 // 分类:OpenWrt // No Comments

Centos服务器安装shadowsocks:

yum install -y gcc automake autoconf libtool make build-essential autoconf libtool 
yum install -y curl curl-devel zlib-devel openssl-devel perl perl-devel cpio expat-devel gettext-devel
/usr/local/src/
git clone https://github.com/madeye/shadowsocks-libev.git
cd shadowsocks-libev/
./configure --prefix=/usr/local/shadowsocks
make && make install
cp rpm/SOURCES/etc/init.d/shadowsocks /etc/init.d/
chmod +x /etc/init.d/shadowsocks 

配置文件:

/etc/shadowsocks/config.json 
{
    "server":"116.251.216.87",
    "server_port":443,
    "local_port":1443,
    "password":"passwd",
    "timeout":600,
    "method":"aes-256-cfb"
}

官方有提供OpenWRT shadowsocks客户端,不过是通过开发版编译的, 稳定版使用出现以下错误:

ss-local: can't resolve symbol 'epoll_create'

自编译shadowsocks:

useradd openwrt
su -l openwrt
svn co svn://svn.openwrt.org/openwrt/tags/backfire_10.03.1
./scripts/feeds update -a
./scripts/feeds install -a
cd package/
git clone https://github.com/madeye/shadowsocks-libev.git
cd ../
make menuconfig

选择平台,Network菜单选择要编译的软件包shadowsocks-libev。
开始编译:

make V=99

查看编译的软件包:

ls bin/brcm63xx/packages/

配置:

/etc/shadowsocks.json 
{
    "server":"116.251.216.87",
    "server_port":443,
    "local_port":1443,
    "password":"passwd",
    "timeout":600,
    "method":"aes-256-cfb"
}
分类
最新文章
最近回复
  • 常易北: 满分顶楼主!!!我也是,现在网上瞎几把扫别人云主机的智障太多了!
  • track: NAT 部分没处理,这块感觉有问题。
  • yu: 查到有用的资料很幸福!
  • boscotsang: 请问下服务器端如何控制权限让通过权限认证的客户端才能连接?现在这样只要有自编译的客户端程序都可...
  • yck932: 大侠,请问哪里有 tunnels 详细参考资料,谢谢
  • bubble: 现在linux 4.9以上的内核有个bbr的参数,用这个bbr比用锐速效果更好,可参考我的博文...
  • wang: openwrt 只有 tmp/etc/miniupnpd.conf 修改后又重置了,其他路径找...
  • wbq: 可以了 只是https透明代理如何做呢?
  • wbq: 你的这个配置需要浏览器设置代理ip+port, 现在假设不设置,只配置客户端网关为Nginx那...
  • wbq: 您好,想用Nginx做个透明代理,客户端机器配置下网关(即Nginx那台机器),客户端dns配...