Jade Dungeon

sbt

配置

国内镜像配置

配置在文件：~/.sbt/repository 官方Document看这里：http://www.scala-sbt.org/0.13.2/docs/Detailed-Topics/Library-Management.html#override-all-resolvers-for-all-builds

配置国内代理库。感谢 OSChina 提供了 Maven Center 的镜像, 配置添加它有助于提升下载速度. 

[repositories]
  local
  oschina:http://maven.oschina.net/content/groups/public/

若你知道其他更快的镜像库, 同上配置. 一般互联网企业部署了供内部使用的镜像库(如 nexus ), 也可以配置于此. 阿里云也提供了镜像，所说比oschina的要更快。 

[repositories]
  local
  aliyun: http://maven.aliyun.com/nexus/content/groups/public/

兼容 Ivy 路径布局。大多数中心仓库(repository)是 Maven 的路径布局, 这就导致 SBT 的插件和部分 Ivy 依赖无法从其下载. 

[repositories]
  local
  oschina:http://maven.oschina.net/content/groups/public/ 
  oschina-ivy:http://maven.oschina.net/content/groups/public/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  aliyun: http://maven.aliyun.com/nexus/content/groups/public/
  aliyun-ivy: http://maven.aliyun.com/nexus/content/groups/public/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  huaweicloud-maven: https://repo.huaweicloud.com/repository/maven/
  huaweicloud-ivy: https://repo.huaweicloud.com/repository/ivy/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]

然，对于Spark项目而言，并卵！因为Spark单独定义了项目的构建规则，并不会加载默认的仓库解析配置。

我们需要修改Spark源码下的project/SparkBuild.scala这个文件，参考我下面这个patch进行修改： 

--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -214,7 +214,8 @@ object SparkBuild extends PomBuild {

     // Override SBT's default resolvers:
     resolvers := Seq(
-      DefaultMavenRepository,
+      //DefaultMavenRepository,
+      Resolver.url("aliyun", url("http://maven.aliyun.com/nexus/content/groups/public/")),
       Resolver.mavenLocal,
       Resolver.file("local", file(Path.userHome.absolutePath + "/.ivy2/local"))(Resolver.ivyStylePatterns)
     ),

用Resolver.url("aliyun", url("Index of /groups/public"))替换掉DefaultMavenRepository即可。

提速Maven构建速度

Maven的就比较简单了，是非常通用的方法，我们仅需要修改pom.xml文件即可： 

--- a/pom.xml
+++ b/pom.xml
@@ -228,7 +228,8 @@
       <id>central</id>
       <!-- This should be at top, it makes maven try the central repo first and then others and hence faster dep resolution -->
       <name>Maven Repository</name>
-      <url>https://repo1.maven.org/maven2</url>
+      <!--<url>https://repo1.maven.org/maven2</url>-->
+      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
       <releases>
         <enabled>true</enabled>
       </releases>

配置下面还有个Maven的插件，也参考这个修改掉即可。

精简 url 列表

远程仓库越多越全, 可以基本避免下载不到的问题. 但是, 也可能让下载的时间更长, 让你不愿在等待而放弃下载.

因为, 下载过程中 SBT 会串行的 "询问" 列表中所有的远程仓库, 无论是否找得到.

当依赖树越大, 整个下载的过程就更漫长. 若再遇到响应慢的仓库, 情况恶化的令人发指.

推荐列表策略是:

  1. 本地仓库
  2. 国内(或内网)镜像仓库
  3. 国外官方仓库, 通常#注释掉, 待上面不管用时, 去掉#再做尝试

例子

[repositories]  
  local
  maven-local: file:///home/jade/.m2/repository
  maven-local-ivy: file:///home/jade/.m2/repository, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  aliyun: http://maven.aliyun.com/nexus/content/groups/public/
  aliyun-ivy: http://maven.aliyun.com/nexus/content/groups/public/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  oschina:http://maven.oschina.net/content/groups/public/ 
  oschina-ivy:http://maven.oschina.net/content/groups/public/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  maven-central
#  activator-launcher-local: file://${activator.local.repository-${activator.home-${user.home}/.activator}/repository}, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
#  activator-local: file://${activator.local.repository-/opt/morganstudio/lib-java/activator-dist-1.3.5/repository}, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
#  typesafe-releases: http://repo.typesafe.com/typesafe/releases
#  typesafe-ivy-releasez: http://repo.typesafe.com/typesafe/ivy-releases, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]

要想彻底解决下载包很慢的问题，最好的方法是在自己的电脑上搭建nexus私服， 然后配置maven和ivy的代理仓库（proxy repository）。 用sbt或者maven编译一个项目，第一次执行的时候，会自动下载依赖包到自己的私服上， 以后再有相同的包需要下载，只要从私服下载即可，速度就很快了， 其他机器也可以使用这个nexus私服。 下面是我的配置脚本，在ubuntu 12.04 64bit上用的，开发java/maven或者scala/sbt项目， 都没什么问题。 

#安装maven
sudo apt-get install maven

sudo vi /etc/profile.d/maven.sh
export MAVEN_OPTS="-Xmx8192m -XX:MaxPermSize=4096m"

source /etc/profile.d/maven.sh

#安装nexus私服
cd ~/software
wget http://www.sonatype.org/downloads/nexus-latest-bundle.tar.gz
tar xvzf nexus-latest-bundle.tar.gz
rm nexus-latest-bundle.tar.gz

vi ~/.bash_profile
export NEXUS_HOME=$HOME/software/nexus-<version>
export PATH=$NEXUS_HOME/bin:$PATH

vi ~/.bashrc
[[ -f $HOME/.bash_profile ]] && . $HOME/.bash_profile

source ~/.bash_profile

$NEXUS_HOME/bin/nexus start &
tail -f $NEXUS_HOME/logs/wrapper.log

#登录nexus管理页面
http://<hostname>:8081/nexus
username:admin
password:admin123

#给nexus添加maven和ivy代理仓库
http://download.java.net/maven/2/
http://repo.typesafe.com/typesafe/releases/
http://repo.typesafe.com/typesafe/snapshots/
http://repo.typesafe.com/typesafe/ivy-releases/
http://repo.typesafe.com/typesafe/ivy-snapshots/
http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/
http://oss.sonatype.org/content/repositories/snapshots

#设置maven代理仓库
#桌面版ubuntu
vi ~/.m2/settings.xml
#服务器版ubuntu
sudo vi /etc/maven/settings.xml

<settings>
  <mirrors>
    <mirror>
      <id>nexus</id>
      <mirrorOf>*</mirrorOf>
      <url>http://<hostname>:8081/nexus/content/groups/public</url>
    </mirror>
  </mirrors>
  <profiles>
    <profile>
      <id>nexus</id>
      <repositories>
        <repository>
          <id>public</id>
          <url>http://<hostname>:8081/nexus/content/groups/public</url>
          <releases>
            <enabled>true</enabled>
          </releases>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>public</id>
          <url>http://<hostname>:8081/nexus/content/groups/public</url>
          <releases>
            <enabled>true</enabled>
          </releases>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>
  <activeProfiles>
    <activeProfile>nexus</activeProfile>
  </activeProfiles>
  <servers>
    <server>
      <id>nexus-releases</id>
      <username>admin</username>
      <password>admin123</password>
    </server>
    <server> 
      <id>nexus-snapshots</id>
      <username>admin</username>
      <password>admin123</password>
    </server>
  </servers>
</settings>

#安装scala
cd /opt
sudo wget http://www.scala-lang.org/files/archive/scala-2.10.3.tgz
sudo tar xfz scala-2.10.3.tgz
sudo chown -R `whoami`:root /opt/scala-2.10.3/
sudo chmod -R u+x /opt/scala-2.10.3/bin/

sudo vi /etc/profile.d/scala.sh
export SCALA_HOME=/opt/scala-2.10.3
export PATH=$SCALA_HOME/bin:$PATH

source /etc/profile.d/scala.sh

#安装sbt
cd /opt
sudo wget http://repo.scala-sbt.org/scalasbt/sbt-native-packages/org/scala-sbt/sbt/0.13.1/sbt.tgz
sudo tar xfz sbt.tgz
sudo mv sbt sbt-0.13.1
sudo chown -R `whoami`:root /opt/sbt-0.13.1/
sudo chmod -R u+x /opt/sbt-0.13.1/bin/

sudo vi /etc/profile.d/scala.sh
export SBT_HOME=/opt/sbt-0.13.1
export PATH=$SBT_HOME/bin:$PATH

source /etc/profile.d/scala.sh

sudo $SBT_HOME/bin/sbt
sudo ls -al /root/.sbt/boot/scala-2.10.3/org.scala-sbt/sbt/0.13.1
sudo ls -al /root/.ivy2/cache

mkdir ~/.sbt/0.13/plugins

vi ~/.sbt/0.13/plugins/plugins.sbt
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.4.0")

#设置sbt代理仓库
vi ~/.sbt/repositories
[repositories]
  local
  ivy-proxy-releases: http://<hostname>:8081/nexus/content/groups/public/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  maven-proxy-releases: http://<hostname>:8081/nexus/content/groups/public/

#结束

构建时调用maven已经下载好的库

resolvers += "Local Maven Repository" at "file://"+Path.userHome.absolutePath+"/.m2/repository"

经过我不断的尝试，给出一些方法

  1. 有VPN直接上VPN。
  2. sbt和maven类似，也会下载工具插件，默认目录是用户目录下的.ivy2/cache目录, 这相当于maven的仓库,可以这样理解。 可以去Maven Repository: Search/Browse/Explore 搜索需要的sbt插件手到下载并复制到相对应的cache里的目录。
  3. 在已有的maven项目或新建个空的maven项目，在pom里添加sbt依然，
  4. 让maven去下,最后复制到ivy2仓库里。

昨天研究了一段时间，发现修改sbt配置文件里的仓库地址没有效果， 实际要修改sbt-launch.jar中的sbt目录里的/sbt.boot.properties配置文件, 默认配置如下: 

[scala]
  version: ${sbt.scala.version-auto}

[app]
  org: ${sbt.organization-org.scala-sbt}
  name: sbt
  version: ${sbt.version-read(sbt.version)[0.13.8]}
  class: ${sbt.main.class-sbt.xMain}
  components: xsbti,extra
  cross-versioned: ${sbt.cross.versioned-false}
  resources: ${sbt.extraClasspath-}

[repositories]
  local
  typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
  maven-central

[boot]
  directory: ${sbt.boot.directory-${sbt.global.base-${user.home}/.sbt}/boot/}

[ivy]
  ivy-home: ${sbt.ivy.home-${user.home}/.ivy2/}
  checksums: ${sbt.checksums-sha1,md5}
  override-build-repos: ${sbt.override.build.repos-false}
  repository-config: ${sbt.repository.config-${sbt.global.base-${user.home}/.sbt}/repositories}
© 2015 jade-dungeon.net. Valid CSS & XHTML
Website template by Arcsin